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ABSTRACT 


This  document  describes  the  results  of  the  second  Field  Test  of  Navy  system-related 
Technical  Information  (TI)  developed  by  the  Navy  Technical  Information  Presentation 
System  (NTIPS).  Three  types  of  experimental  TI  were  compared  with  the  conventional 
Technical  Manual.  For  performing  troubleshooting  tasks,  the  NTIPS  electronically 
displayed  automated  troubleshooting  TI  (called  Fault  Isolation  by  Nodal  Dependency, 
FIND)  was  compared  with  the  conventional  Technical  Manual,  and  for  performing 
corrective  maintenance,  NTIPS  electronically  displayed  TI  was  compared  with  NTIPS  on 
paper  and  the  conventional  Technical  Manual. 

Tests  were  carried  out  at  the  Naval  Sea  Combat  Systems  Engineering  Station  in 
Norfolk,  Va.  using  two  operational  AN/SPA-25D  radar  repeaters  (with  introduced  faults). 
Test  subjects  were  Electronic  Technicians  stationed  on  ships  and  at  shore  based  facilities 
in  Norfolk,  Va. 

All  test  objectives  were  achieved.  Almost  all  (92%)  of  the  subjects  preferred 
electronically  delivered  TI  to  the  conventional  Technical  Manual.  They  were  able  to  use 
NTIPS  TI  to  troubleshoot  more  accurately  and  with  greater  speed  than  with  the 
conventional  Technical  Manual.  For  corrective  maintenance,  subjects  performed  at  the 
same  accuracy  and  speed  regardless  of  the  TI  type. 
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1.0  TEST  SUMMARY 


1.1  BACKGROUND  AND  PURPOSE 

This  is  the  second  of  two  operational  field  tests  of  the  Navy  Technical  Information 
Presentation  System  (NTIPS),  a  system  that  has  been  designed  to  improve  the  quality  and 
reduce  the  difficulty  and  expense  of  acquiring  and  managing  Technical  Information  (TI) 
for  logistic  support  of  Navy  weapons  systems.  NTIPS  maximizes  reliance  on  automated 
systems,  starting  with  documentation  specifications  and  authoring  procedures  and 
culminating  in  the  electronic  delivery  of  TI  to  the  technician  at  the  maintenance  site. 
NTIPS  is  currently  in  Phase  III:  Test  and  Evaluation.  Both  tests  were  designed  to 
compare  TI  generated  using  NTIPS  procedures  with  conventional  paper  technical  manuals 
used  for  performing  troubleshooting  and  corrective  maintenance.  The  test  tasks  in  the 
first  test  were  performed  on  the  Rudder  Manual  Trim  System  of  the  F-14A  at  Naval  Air 
Station,  Miramar,  California.*  The  SPA-25D  radar  repeater  was  used  in  the  second  test 
which  was  conducted  at  the  Naval  Sea  Combat  Systems  Engineering  Station,  Norfolk, 
Virginia.  Both  systems  were  selected  by  NTIPS  staff  from  a  list  of  candidates  prepared  by 
the  Naval  Technical  Manual  Management  Policy  Council. 

During  the  spring  of  1987,  a  test  plan  was  developed+  for  the  field  test  using  the 
AN/SPA-25D  radar  repeater.  A  pretest  of  the  plan  was  conducted  in  June  1987  at  the 
Naval  Sea  Combat  Systems  Engineering  Station  and  the  plan  was  modified  in  August  1987. 
The  full  test  was  conducted  in  September.  The  objectives  of  the  field  test  were  as 
follows: 

o  Compare  the  performance  of  enlisted  maintenance  technicians  using 
the  TI  prepared  under  NTIPS  procedures  with  the  performance  of 
technicians  using  conventional  TI  (the  conventional  paper  Technical 
Manual  for  the  AN/SPA-25D). 

o  Compare  technicians'  performance  when  guided  by  TI  printed  on 
paper  versus  TI  presented  via  an  electronic  medium. 


*  Test  and  evaluation  of  the  Navy  Technical  Information  Presentation  System  (NTIPS): 
F-14A  Experimental  Technical  Information  Field  Test.  June  1987  (Essex 
Corporation) 

+  NAVY  TECHNICAL  INFORMATION  PRESENTATION  PROGRAM.  Phase  III  Test 
and  Evaluation  of  the  Navy  Technical  Information  Presentation  System.  AN/SPA- 
25D  Experimental  Technical  Information  Test  Plan.  August  1987  (Essex  Corpora¬ 
tion) 
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o  Establish  which  design  characteristics  of  NTIPS  TI  are  most  effective 
or  least  effective  in  an  operational  situation. 

o  Assess  user  acceptance  of  certain  features  of  the  NTIPS  (medium, 
content,  format,  and  style)  TI  presentation. 

All  of  *'  ese  objectives  were  achieved.  This  section  provides  an  overview  of  the  test 
design,  test  execution,  and  test  results.  This  field  test  of  NTIPS  TI  was  performed  using 
off-the-shelf  electronic  delivery  devices.  The  test  was  not  designed  to  evaluate  fielded 
hardware,  but  rather  to  test  NTIPS  approaches  to  creating  and  electronically  displaying  TI 
that  is  expected  to  be  intrinsically  more  effective  than  conventional  TI,  thus  reducing 
fleet  reliance  on  paper  manuals.  In  addition,  the  test  was  designed  to  provide  guidance  in 
establishing  areas  of  needed  improvement  to  the  TI  and  the  TI  electronic  delivery  device, 
as  well  as  to  demonstrate  the  current  effectiveness  of  the  NTIPS  approaches. 

1.2  TEST  SITE  AND  TEST  PERSONNEL 

The  field  test  was  conducted  at  Naval  Sea  Combat  Systems  Engineering  Station, 
Norfolk,  Virginia  in  the  test  bay  area.  The  test  utilized  two  operational  AN/SPA-25D 
radar  repeaters.  The  test  subjects  were  24  active-duty  Electronic  Technicians  (ETs)  made 
available  by  the  Commander  Naval  Surface  Forces  U.S.  Atlantic  Fleet 
(COMNAVSURFLANT).  For  test  purposes  these  technicians  were  assigned  to  experienced 
and  inexperienced  groups,  based  on  the  length  of  time  they  had  spent  performing 
maintenance  on  radar  systems  and  other  electronic  equipment;  those  with  over  one  year 
of  experience  were  assigned  to  the  experienced  group,  the  others  were  assigned  to  the 
inexperienced  group.  There  were  11  experienced  technicians  and  13  inexperienced 
technicians. 

In  addition  to  test  subjects,  the  following  personnel  were  required  for  the  test:  two 
test  coordinators  from  the  Combat  System  Engineering  Station;  a  Test  Director,  two 
computer  analysts,  and  a  video  crew  from  David  Taylor  Research  Center;  a  computer 
specialist  from  Hughes  Aircraft;  and  three  data  collectors  from  Essex  Corporation. 

1.3  SCOPE  OF  THE  TEST 

The  field  test  was  conducted  using  two  operationally  realistic  troubleshooting  tasks 
and  three  corrective  maintenance  tasks.  As  the  basis  for  the  troubleshooting  tasks,  faults 
were  introduced  into  the  AN/SPA-25D  repeaters  by  the  test  coordinators.  For  each  fault, 
test  subjects  were  asked  first  to  verify  the  fault  and  then  to  isolate  it.  On  Fault  1,  half  of 
the  test  subjects  used  the  electronic  display  system  (Fault  Isolation  by  Nodal 
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Dependency  -  FIND)  and  half  used  the  conventional  Technical  Manual.  Those  who  used 
the  electronic  display  for  Fault  1  used  the  conventional  Technical  Manual  for  the  task 
involving  isolation  of  Fault  2  and  vice  versa.  FIND  is  an  automated  troubleshooting 
procedure  designed  to  lead  the  technician  through  fault  isolation.  All  subjects  performed 
three  corrective  maintenance  tasks,  each  with  a  different  type  of  TI:  NTIPS  presented 
electronically,  NTIPS  in  a  paper  medium,  and  the  conventional  Technical  Manual.  The 
assignment  of  TI  type  to  corrective  maintenance  tasks  and  the  order  of  task  assignment  to 
subjects  was  counterbalanced  across  the  subjects. 

Effectiveness  of  the  types  of  troubleshooting  and  corrective  maintenance  TI  was 
compared  by  evaluating  the  performance  of  technicians  using  each  type  of  TI. 
Performance  measures  included  the  time  required  for  successful  completion  of  the  task 
and  the  number  of  errors  committed  by  technicians  during  task  performance.  These 
measures  were  supplemented  by  the  subjects'  own  evaluations,  obtained  by  questionnaire 
and  interview,  of  the  operational  justifiability,  usefulness,  and  effectiveness  of  each  type 
of  TI  for  performing  troubleshooting  or  corrective  maintenance. 


1.4  TEST  EVENTS 

Once  preparation  of  the  experimental  TI  was  completed,  the  field  test  consisted  of 
the  following  events: 

1.  Dry  Run,  May  1987.  The  proposed  Test  tasks  and  the  experimental  TI 
were  checked  by  senior  technicians  from  Clifton  Precision,  Incorporated, 
manufacturer  of  the  AN/SPA-2 5 D,  on-site  at  the  Naval  Sea  Combat 
Systems  Engineering  Station. 

2.  Pretest,  29  June-1  July  1987.  During  this  event,  each  of  four  ETs 
performed  one  of  the  troubleshooting  tasks  and  either  the  three  removal 
or  the  three  replacement  tasks  used  in  the  Test  to  evaluate  the 
experimental  TI  generated  for  the  Field  Test.  All  three  types  of  TI  were 
checked.  As  a  result  of  this  event,  a  number  of  changes  were  made  to 
the  NTIPS  TI  and  to  the  test  plan. 

3.  Field  Test,  8-24  September  1987.  The  performance  of  24  ETs  was 
measured  while  they  performed  troubleshooting  and  corrective  mainte¬ 
nance  tasks  using  NTIPS  TI  and  the  conventional  Technical  Manual.  For 
corrective  maintenance,  the  NTIPS  TI  was  presented  both  electronically 
and  on  paper.  NTIPS  troubleshooting  TI  was  presented  only 
electronically.  Technicians  also  provided  preference  and  rating  informa¬ 
tion  on  a  questionnaire  and  in  a  post-test  debriefing. 


1.5  CONDUCT  OF  THE  FIELD  TEST 

Test  subjects  for  the  Test  were  24  ETs,  11  experienced  and  13  inexperienced.  After 
an  instruction  session  on  the  Test  and  on  the  use  of  the  electronic  delivery  device  for  both 
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troubleshooting  and  corrective  maintenance,  Test  personnel  asked  subjects  to  solve  the 
first  troubleshooting  problem  and  assigned  either  NTIPS  FIND  or  the  conventional 
Technical  Manual  as  the  TI  to  be  used.  Following  completion  of  the  first  problem,  Test 
personnel  gave  the  subjects  a  second  troubleshooting  task,  and  supplied  them  with  the  TI 
type  they  had  not  used  for  the  first  problem.  When  both  troubleshooting  problems  were 
completed,  each  subject  filled  out  an  evaluation  questionnaire  comparing  the  two  types  of 
TI.  The  second  part  of  the  test  involved  the  performance  of  three  corrective  maintenance 
tasks  by  each  technician  using  a  different  type  of  TI  (conventional  Technical  Manual, 
NTIPS  Electronic  Delivery,  and  NTIPS  Paper)  for  each  task.  At  the  end  of  the  test,  an 
evaluation  questionnaire  comparing  the  TI  used  for  corrective  maintenance  was  completed 
by  each  technician,  and  Test  personnel  conducted  an  interview.  During  troubleshooting 
and  corrective  maintenance  performance,  Test  personnel  recorded  both  performance  time 
and  errors  committed  (for  each  step). 


1.6  SUMMARY  OF  TEST  RESULTS 

All  test  objectives  were  accomplished.  Careful  observation  of  technician  perform¬ 
ance  on  several  types  of  tasks  using  conventional  Technical  Manuals  and  NTIPS  TI,  both 
electronic-delivery  and  paper  presentation,  showed  the  following  results: 

1.  For  troubleshooting  tasks.  The  inexperienced  technicians'  performance 
time  was  26%  faster  with  FIND  than  with  the  conventional  Technical 
Manual;  the  experienced  technicians'  performance  time  was  22%  faster 
with  FIND.  These  performance  time  improvements  could  be  increased  if 
the  time  spent  waiting  for  the  system  to  respond  could  be  reduced.  At 
the  present  time,  60%  of  the  technicians'  TI  use  time  is  spent  in  waiting 
for  the  next  screen. 

2.  For  troubleshooting  tasks.  All  technicians  were  given  15  minutes  to 
initiate  some  testing  or  action  judged  to  be  relevant  to  isolating  the 
fault.  Technicians  who  were  not  able  to  do  so  were  told  how  to  start, 
e.g.,  were  given  significant  assistance.  All  technicians  using  FIND 
isolated  the  fault  without  significant  assistance.  Only  58%  of  the 
technicians  using  the  conventional  Technical  Manual  isolated  the  fault 
without  signficant  assistance;  the  remaining  42%  had  to  be  told  to  use 
the  Fault  Logic  Diagram,  a  presentation  which  steps  through  a  complete 
or  partial  fault  isolation. 

3.  For  troubleshooting  tasks.  Some  technicians  needed  minor  help  while 
performing  work  relevant  to  isolating  the  fault.  Test  personnel 
prompted  these  technicians  (e.g.,  helped  interpret  a  waveform)  as 
required.  The  number  of  prompts  provided  to  technicians  when  using 
FIND  was  7;  when  using  the  conventional  Technical  Manual,  39.  When 
using  the  conventional  Technical  Manual,  experienced  technicians  were 
given  15  prompts  and  inexperienced  technicians  were  given  24  prompts. 
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4.  For  corrective  maintenance  tasks.  Performance  times  for  all  three 
types  of  TI  (NTIPS  electronically  delivered,  NTIPS  on  paper,  and 
conventional  Technical  Manual)  were  essentially  the  same.  Additionally, 
there  were  no  significant  differences  between  performance  times  of 
experienced  and  inexperienced  technicians. 

5.  For  corrective  maintenance  tasks.  Inexperienced  technicians  committed 
the  greatest  number  of  errors  when  using  the  conventional  Technical 
Manual;  experienced  technicians  committed  the  greatest  number  of 
errors  when  using  NTIPS  delivered  electronically.  Eighty-seven  percent 
of  these  errors  made  by  experienced  technicians  can  be  attributed  to 
inadequate  graphics.  Experienced  technicians  made  23%  fewer  errors 
than  inexperienced  technicians. 

6.  Technician  preference.  All  24  technicians  preferred  FIND  to  the 
conventional  Technical  Manual  for  troubleshooting.  The  primary  reasons 
given  were  the  step-by-step  text  instructions  and  the  integration  of  text 
and  graphics  in  FIND.  For  the  step-by-step  text  instructions  in  the 
corrective  maintenance  TI,  63%  of  the  technicians  preferred  NTIPS 
electronic  delivery,  17%  preferred  NTIPS  on  paper  and  20%  preferred 
the  conventional  Technical  Manual.  Experienced  technicians  preferred 
the  graphics  presented  in  NTIPS  paper  and  inexperienced  technicians 
preferred  the  conventional  Technical  Manual  graphics. 


1.7  TEST  CONCLUSIONS  AND  TECHNICIAN  RECOMMENDATIONS 

With  the  cooperation  of  COMNAVSURFLANT  and  the  Naval  Sea  Combat  Systems 
Engineering  Station,  the  NTIPS  Field  Test  achieved  all  test  objectives.  Based  on  Test 
data  the  following  conclusions  have  been  reached: 

1.  Technical  Information  presented  electronically  represents  a  distinct 
improvement  in  the  eyes  of  fleet  technicians  engaged  in  maintenance  of 
electronic  equipment.  This  result  applied  particularly  to  troubleshooting 
and  was  obtained  even  with  the  use  of  off-the-shelf,  non-portable 
computers  which  were  not  optimally  designed  for  operational  use  and 
with  graphics  which  were  clearly  in  need  of  much  improvement. 

2.  NTIPS  presentations  of  troubleshooting  TI  resulted  in  significant 
improvements  in  fault-isolation  effectiveness  and  performance  time. 

3.  NTIPS  presentations  of  corrective-maintenance  TI  did  not  lead  to 
significantly  reduced  test  performance  time  of  either  experienced  or 
inexperienced  technicians.  For  inexperienced  technicians,  the  number  of 
errors  committed  during  the  corrective  maintenance  procedures  were 
reduced. 

4.  Two  major  weaknesses  were  identified  in  the  NTIPS  TI.  The  graphic 
presentations  in  the  corrective  maintenance  TI  were  difficult  for 
technicians  to  use  in  identifying  and  locating  the  parts  called  out  in  the 
step-by-step  text  instructions.  The  system’s  response  time  for  the  NTIPS 
troubleshooting  was  slow.  Improvements  in  these  two  areas  might 
significantly  increase  the  usability  of  NTIPS  TI. 
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Test  personnel  and  subject  technicians  provided  valuable  recommendations  during 
the  test.  For  example:  it  was  made  clear  that  the  quality  of  graphics  in  any  future  uses 
of  electronic  presentation  must  be  much  improved  over  that  of  the  graphics  used  in  the 
test.  This  improvement  would  involve  modifications  in  the  current  design  of  the  graphic, 
in  graphic  size,  and  in  graphic  resolution.  Technicians'  comments  also  demonstrated  the 
need  for  greater  flexibility  in  the  automated  TI  presentations  to  permit  experienced 
technicians  to  move  more  rapidly  through  a  series  of  steps  without  the  time-consuming 
necessity  of  continually  viewing  material  they  already  know  from  experience. 
Technicians  also  suggested  use  of  animation  to  portray  the  correct  waveform  for 
troubleshooting. 

Detailed  results  of  the  tests  and  specific  recommendations  of  the  test  personnel  are 
discussed  in  Section  4.0. 

1.8  ORGANIZATION  OF  TEST  REPORT 

Section  2  describes  the  preparation  and  review  of  the  experimental  TI  (electronic 
presentation  and  paper)  used  for  the  test.  Section  3  describes  the  actual  tasks  selected 
for  testing  and  summarizes  the  Test  design,  which  was  independently  published  as  a  DTRC 
report.*  As  described  in  Section  3,  the  initial  Test  Plan  was  modified  to  some  extent  as  a 
result  of  the  Pretest  Trials  discussed  previously.  Section  4  discusses  in  detail  the  test 
results  and  summarizes  performance  times,  performance  errors,  and  other  results 
obtained  by  monitoring  the  Test  task  performances  of  technicians  using  the  five  kinds  of 
TI  tested.  Section  4  also  includes  the  subjects'  preferences  and  their  recommendations  for 
improving  NTIPS  TI  and  its  electronic  presentation.  Section  5  summarizes  the  Field  Test 
and  presents  overall  conclusions.  Appendix  A  contains  the  actual  forms  used  for  data 
collection.  Appendix  B  shows  the  preference  questionnaire  administered  to  the  24  Test 
subjects.  Appendix  C  contains  samples  of  the  TI  tested. 

*  NAVY  TECHNICAL  INFORMATION  PRESENTATION  PROGRAM.  Phase  III  Test 
and  Evaluation  of  the  Navy  Technical  Information  Presentation  System.  AN/SPA- 
25D  Experimental  Technical  Information  Test  Plan.  August  1987  (Essex  Corpora¬ 
tion) 
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2.0  PREPARATION  AND  APPROVAL  OF  EXPERIMENTAL  AND  CONVENTIONAL  TI 


This  section  discusses  the  generation  of  the  experimental  NTIPS  TI  used  for  the  Test 
(2.1),  the  review  of  this  TI  for  test  suitability  (2.2),  and  the  review  of  conventional  TI  for 
compatibility  with  the  NTIPS  TI  (to  ensure  that  it  covered  the  same  tasks)  (2.3). 

2.1  GENERATION  OF  EXPERIMENTAL  TI 

The  experimental  TI  was  prepared  by  Clifton  Precision  Incorporated,  manufacturer 
of  the  AN/SPA-25D  Radar  Repeater.  In  order  to  prepare  the  TI  for  test  purposes,  Clifton 
Precision  was  provided  a  MODCOMP  computer  which  hosted  the  NTIPS  prompted 
automated  authoring  system,  a  terminal,  a  screen  printer,  and  a  modem  which  permitted 
communication  with  a  similar  MODCOMP  at  Hughes  Aircraft  Company  for  obtaining 
assistance  when  problems  arose.  Clifton  Precision  personnel  were  trained  by  Hughes  in 
operation  of  the  authoring  system  and  in  inserting  system  signal-dependency  information 
into  the  FIND  automated  troubleshooting  program. 

NTIPS  specifications  provided  to  Clifton  Precision  for  use  in  preparing  the 
experimental  TI  included  general  content,  format,  and  style  specifications  covering  the 
following  TI  characteristics: 

a.  Procedures 

b.  Descriptive  information 

c.  Illustrations 

d.  Style  (general) 

e.  Numbering,  indexing,  and  how-to-use  information 

f.  Diagrams. 

Also  provided  was  the  NTIPS  specification  entitled  "Fault  Isolation  by  Nodal  Dependency 
(FIND):  Troubleshooting  Equipment,  Software,  and  Products"  to  permit  generation  of 
electronically  displayed  troubleshooting  information.  FIND  is  an  interactive  system  that 
selects  a  series  of  optimal  test  points  based  on  a  network  of  component 
interdependencies,  component  failure  rates,  and  time-to-test  requirements.  The 
technician  enters  fault  symptoms  which  the  FIND  software  uses  to  identify  the  first  test 
point;  the  test  results  from  the  first  test  provide  FIND  with  the  information  to  select  the 
next,  most  logical  test  point.  This  sequence  continues  until  FIND  isolates  the  fault. 
Procedural  instructions  and  supporting  graphics  are  provided  to  the  technician  for  each 
test  point. 
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The  above  specifications  are  all  draft  documents  prepared  under  NTIPS  and  are  now 
being  circulated  among  the  System  Commands  for  comment  and  possible  adoption. 

Before  delivery  of  the  experimental  TI  was  made  to  NTIPS,  Clifton  Precision 
personnel  validated  the  draft  TI  with  the  use  of  an  operational  AN/SPA-25D  made 
available  by  Naval  Sea  Combat  Systems  Engineering  Station.  Validation  consisted  of  a 
technical-accuracy  review  by  subject-matter  experts  based  on  comparing  the  NTIPS  TI 
against  the  AN/SPA-25D  hardware.  Reviews  to  ensure  that  the  experimental  TI  was  in 
compliance  with  NTIPS  specifications  for  content  and  style  were  performed  by  Clifton 
Precision,  by  Hughes  Aircraft  Company,  and  by  the  NTIPS  office. 

Observations  made  by  Clifton  Precision  during  preparation  of  the  experimental  TI 
were  recorded  in  detail  in  a  Journal  format  (a  Log),  and  these  data  will  be  evaluated  to 
establish  the  possible  need  for  NTIPS  modifications.  Certain  changes  to  the  NTIPS- 
designed  authoring  system  and  to  the  electronic-display  system  have,  in  fact,  alrendy  been 
carried  out  as  a  result  of  Clifton's  experience. 

Experimental  TI  was  generated  for  the  following  AN/SPA-25D  troubleshooting  and 
corrective  maintenance  tasks: 

o  A  brief  fault  verification  procedure  in  a  text  format.  Verification  of 
a  fault  produces  the  fault  symptoms  required  for  entering  the  fault 
isolation  procedure. 

o  Troubleshooting  TI  implemented  on  the  FIND  system.  This  TI  was 
used  to  isolate  the  faulty  component  producing  a  failure  in  (1)  the 
sweep  resolver  and  (2)  the  main  gate  generator.  The  only  delivery 
medium  for  FIND  TI  is  electronic  display  using  (in  this  test)  a 
cathode-ray  tube. 

o  Corrective-maintenance  procedures  (both  electronic  display  and  on 
paper)  for  the  replacement  (removal  of  a  component  and  installation 
of  a  new  component)  and  check  and  adjustment  of  the  focusing  coil 
and  azimuth  resolver  baseplate. 

o  TI  procedures  (both  electronic  display  and  on  paper)  for  all  supporting 
tasks  involved  in  readying  the  AN/SPA-25D  for  maintenance,  such  as 
indicator  setup  for  maintenance,  restoring  the  radar  repeater  to  a 
ready  condition,  and  an  illustrated  parts  breakdown  (IPB)  for  relevant 
parts. 

In  accordance  with  NTIPS  procedures,  Clifton  Precision  generated  the  experimental 
TI  in  a  single  electronic  data  base,  which  was  output  in  both  paper  and  electronic-delivery 
format  by  Hughes  Aircraft  during  a  mastering  process.  In  the  Test,  NTIPS 
troubleshooting  TI  was  delivered  electronically  by  an  AT&T  3B-2/300  computer  with  an 
AT&T  PC  6300  serving  as  the  subject's  interactive  terminal.  The  subject  entered 
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commands  via  the  6300's  touch  screen  or  keyboard.  The  terminal  weighs  approximately  40 
pounds.  The  cathode-ray  display  (green)  of  the  6300  terminal  has  a  screen  size  of  9.5  x  7 
inches  and  a  resolution  of  640  x  400  lines.  For  corrective  maintenance,  a  Zenith  248 
personal  computer  was  used.  The  Zenith  monitor  is  8"  X  10",  has  a  resolution  of  640  X 
350  lines,  and  has  a  color  display.  The  terminal  weighs  approximately  40  pounds.  When 
implemented,  the  NTIPS  display  system  will  consist  of  a  single,  militarized  portable 
device  which  will  weigh  approximately  10  pounds,  with  dimensions  of  12"  x  9"  x  2",  a 
screen  size  of  6.4"  x  9.6",  and  a  resolution  of  640  lines  x  960  lines. 

Except  for  FIND  troubleshooting  TI,  the  TI  described  above  was  provided  in  paper 
form  as  well  as  electronic-delivery  form. 

2.2  UPGRADE  OF  NTIPS  TI 

The  evaluation  of  the  NTIPS  TI  for  the  AN/SPA-25D  is  the  second  of  two  Field 
Tests,  the  first  being  conducted  at  the  Miramar  Naval  Air  Station,  San  Diego,  CA  on  the 
F-14A  Aircraft.  Lessons  learned  from  the  F-14A  Test  were  applied  to  upgrade  the  NTIPS 
TI  for  the  AN/SPA-25D.  Two  major  improvements  to  the  AN/SPA-25D  TI  concerned  the 
locator  graphics  and  test  set-up  instructions  in  the  FIND  TI,  and  the  time  required  for  the 
system  to  display  corrective  maintenance  TI.  Each  improvement  is  summarized  below. 

a.  FIND  Text  Graphics.  Observation  of  the  fault  isolation  performance  during 
the  F-14A  Test  and  during  the  AN/SPA-25D  Pretest  indicated  improvement  possibilities 
for  two  aspects  of  the  FIND  TI:  (1)  text  instructions  for  setting  up  test  equipment,  the 
oscilloscope  in  the  case  of  the  AN/SPA-25D;  and  (2)  graphics  to  aid  the  technician  in 
performing  tests.  For  the  AN/SPA-25D  these  improvements,  developed  by  technical 
personnel  at  the  DTRC,  involved  card  locators,  test  point  locators,  and  waveform  graphics 
(See  Appendix  C). 

b.  System  Response  Time  for  Displaying  TI.  The  preference  data  collected  from 
the  technicians  participating  in  the  F-14A  NTIPS  Test  indicated  a  strong  objection  to  the 
long  waits  for  the  NTIPS  device  to  display  its  TI.  Technical  personnel  at  the  DTRC 
rehosted  corrective  maintenance  TI  to  a  Zenith  248  computer  and  its  display  terminal. 
This  device  provided  response  times  which  were  considerably  faster  than  the  device  used 
to  deliver  NTIPS  TI  during  the  F-14A  Field  Test. 

2.3  REVIEW  AND  APPROVAL  OF  EXPERIMENTAL  TI  BY  NTIPS 

The  acceptability  of  the  experimental  TI  was  based  on  a  detailed  review  performed 
by  NTIPS  personnel  and  contractor  organizations  (Hughes  Aircraft  and  Essex  Corporation). 
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This  review  was  supplemented  by  a  final  on-site  validation  at  Naval  Sea  Combat  Systems 
Engineering  Station  in  which  the  procedures  and  graphics  were  checked  by  Clifton 
Precision  personnel  against  an  operational  AN/SPA-25D.  A  review  by  NTIPS  personnel 
established  that  the  draft  TI  complied  with  the  NTIPS  style,  content,  and  format 
specifications.  The  validation  performed  in  the  field  ensured  that  the  experimental  TI 
contained  all  information  needed  by  the  technicians,  that  this  information  was  accurate, 
and  that  it  was  presented  as  clearly  and  simply  as  possible.  Problems  identified  by  these 
reviews  were  documented  and  corrected  by  Clifton  Precision  or  Hughes  Aircraft. 


3.0  TEST  DESIGN 


3.1  TEST  PERSONNEL 

The  NTIPS  AN/SPA-25D  Test  was  conducted  during  the  period  September  8  to 
September  24,  1987  in  close  coordination  with  the  Naval  Sea  Combat  Systems  Engineering 
Station,  Norfolk,  VA.  Actual  performance  of  the  test  tasks  took  place  in  the  test  bay 
area  of  the  Station.  A  listing  of  test  personnel  and  their  functions  is  shown  in  Table  1. 
Test  coordination,  technical  consultation,  equipment  check-out,  and  fault  insertion  were 
provided  by  technical  personnel  of  the  Naval  Sea  Combat  Systems  Engineering  Station. 
Test  subject  selection  and  assignment  was  accomplished  by  Naval  personnel  stationed  at 
COMNAVSURFLANT.  Detailed  test  scheduling  was  performed  by  management  personnel 
at  the  Naval  Sea  Combat  Systems  Engineering  Station.  Twenty-four  technicians  from 
Norfolk-based  ships  or  from  local  shore-based  facilities  participated  as  test  subjects. 
These  subjects  performed  troubleshooting  and  corrective  maintenance  tasks  as  specified 
in  the  Test  Plan  (3.3),  responded  to  a  preference  questionnaire  designed  to  evaluate  the 
strengths  and  weaknesses  of  NTIPS  TI  as  compared  to  conventional  paper  Technical 
Manuals,  and  provided  comments  during  debriefings. 

The  test  was  supervised  by  a  Test  Director  from  David  Taylor  Research  Center. 
The  Test  Director  coordinated  the  Test  with  personnel  from  the  host  station,  provided 
technical  direction  on  test  performance,  and  ensured  that  the  test  was  conducted 
smoothly  and  on  schedule.  The  Test  Director  was  supported  by  (1)  three  computer 
specialists,  two  from  David  Taylor  Research  Center  and  one  from  Hughes  Aircraft 
Company,  (2)  three  data  collectors  from  the  Essex  Corporation,  and  (3)  a  video  camera 
crew  from  David  Taylor  Research  Center.  The  computer  specialists  briefed  subjects  on 
the  use  of  the  electronic  delivery  devices  and  the  NTIPS  TI,  and  assisted  with  the 
operation  of  the  delivery  devices  as  needed  throughout  the  test.  The  data  collectors 
presented  test  briefings  and  debriefings,  recorded  performance  times,  errors,  and  activi¬ 
ties  for  each  maintenance  task,  and  administered  the  preference  questionnaire.  The  video 
crew  taped  samples  of  troubleshooting  and  corrective  maintenance  task  performance  and 
debriefing  sessions  for  12  of  the  subject  technicians. 
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Table  1.  Test  personnel  and  functions. 

Source  of  Test  Personnel  Functions 

Naval  Sea  Combat  Systems  Engineering 
Station 


o  Management  Personnel 
o  Technical  Personnel 

COMNAVSURFLANT 

Naval  Ships  and  Shore-Based  Facilities 
o  24  Electronic  Technicians  as  test 
subjects 

David  Taylor  Research  Center 
o  Test  Director 


o  Coordinated  subjects  and  facilities, 
accomplished  detailed  scheduling 

o  Inserted  faults  for  troubleshooting 
problems 

o  Checked-out  equipment  following  each 
test  performance 

o  Monitored  subject-technician 
performance  for  safety 

o  Arranged  for  subject-technicians  to 
participate  in  the  test 


o  Performed  test  tasks  using 

conventional  Technical  Manual  and 
FIND  for  troubleshooting,  and  the 
conventional  Technical  Manual,  NTIPS 
TI  electronically  delivered,  and  NTIPS 
TI  on  paper  for  corrective 
maintenance 

o  Responded  to  preference  questionnaire 
and  post-test  debriefing 


o  Served  as  principal  NTIPS 
representative 

o  Directed  the  test 

o  Coordinated  test  with 

COMNAVSURFLANT  and  host  station 
personnel 
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Table  1  (Continued) 


Source  of  Test  Personnel 


Functions 


o  Computer  Specialists 

0 

Briefed  subject  technicians  on  the  use 
of  computers  and  the  NTIPS  TI 

0 

Maintained  computers  used  for 
electronic  delivery  of  TI 

0 

Maintained  and  modified  software  as 
needed 

Hughes  Aircraft  Company 
o  Computer  Specialist 

0 

Briefed  subject-technicians  on  the  use 
of  computers  and  NTIPS  TI 

0 

Maintained  computers  used  for 
electronic  delivery 

0 

Maintained  and  modified  software  as 
needed 

Essex  Corporation 
o  Data  Collectors 

0 

Presented  test  briefings  describing  the 
test's  purpose  and  procedures 

0 

Collected  data  on  subject-technician 
activities,  performance  times,  and 
errors 

0 

Conducted  debriefings  of  subjects  and 
administered  preference 
questionnaires 

0 

Analyzed  data  and  prepared  test 
report 
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3.2  MAINTENANCE  TASK  DEFINITION 

Test  subjects  performed  two  troubleshooting  tasks  and  three  corrective  maintenance 
tasks  on  the  AN/SPA-25D  Radar  Repeater  using  different  types  of  Technical  Information. 
The  AN/SPA-25D  is  a  general-purpose  Plan  Position  Indicator  (PPI)  designed  for  remote 
display  of  azimuth  and  range  information  for  targets  detected  by  a  radar  set.  Target 
bearing  is  determined  by  means  of  an  electronic  cursor,  target  range  by  means  of  a  range 
strobe.  The  range  strobe  can  be  used  either  as  a  video  strobe,  appearing  as  a  ring  on  the 
video  sweep,  or  as  a  cursor  strobe  appearing  as  a  brightened  spot  (marker)  on  the 
electronic  cursor.  Range  rings  are  provided  for  estimating  the  range  of  targets  without 
using  the  range  strobe.  The  display  range  of  the  indicator  is  variable  continuously  from  1 
to  300  miles  consistent  with  the  pulse  repetition  frequency  (prf)  of  the  input.  The 
indicator  will  operate  with  prfs  from  10  to  5,000  pulses  per  second  (pps).  The  AN/SPA- 
25D  contains  an  azimuth-range  indicator  with  an  attached  air  cooler.  The  repeater 
includes  four  functional  sections:  timing,  sweep,  brightening,  and  power  supply. 

Two  faults  were  selected  for  the  troubleshooting  tasks;  one  in  the  main  gate 
generator  card  and  the  other  in  the  sweep  resolver.  The  fault  in  the  main  gate  generator 
was  a  disconnected  capacitor  which  resulted  in  a  blank  scope.  The  sweep  resolver  fault, 
introduced  by  taping  over  a  terminal,  led  to  the  disappearance  of  the  North-South  sweep. 
The  sweep  resolver  fault  was  introduced  first,  and  the  subject  technicians  were  directed 
to  proceed  through  fault  verification  and  fault  isolation.  Once  the  sweep  resolver  fault 
was  isolated  (the  faulty  component  determined),  the  first  fault  was  removed  and  the 
second  fault  was  introduced  (lifting  the  capacitor  on  the  main  gate  generator).  The 
subject  technician  then  verified  and  isolated  this  fault. 

The  corrective  maintenance  tasks  selected  for  the  Test  included  removal  and 
installation  of  the  variable  delay  line,  of  circuit  card  assemblies,  and  of  support  brackets. 
These  are  three  of  the  subtasks  involved  in  performing  maintenance  on  the  Tube  Focusing 
Coil.  The  following  criteria  were  used  in  selecting  these  tasks: 

o  Tasks  must  be  capable  of  being  performed  in  an  operational  context 
represented  by  the  maintenance  facilities  of  an  organizational-level 
maintenance  shop. 

o  Tasks  must  be  presented  in  a  way  that  conforms  to  procedures  and 
methods  normally  used  in  an  operational  setting. 

o  Tasks  must  be  capable  of  being  performed  by  the  type  of  technicians 
called  for  in  typical  maintenance  operations. 

o  Performance  of  the  tasks  must  not  require  support  effort  for  which 
NTIPS  TI  is  not  available. 

o  A  task  should  be  neither  too  simple  nor  too  complex.  The  former 
would  provide  inconclusive  results,  and  the  latter  would  require  too 
lengthy  an  overall  test  schedule. 
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Both  the  troubleshooting  and  corrective  maintenance  tasks  selected  for  the  Test 
were  determined  to  be  representative  of  maintenance  tasks  regularly  performed  on  the 
AN/SPA-25D  radar  repeater.  Table  2  presents  a  typical  maintenance  sequence;  the  tasks 
used  in  the  test  were  selected  from  this  sequence. 

3.3  TEST  PLAN  AND  SUBJECTS 

The  Test  was  designed  so  that  comparisons  could  be  made  based  on  the  technicians' 
performance  using  each  type  of  TI.  Five  combinations  of  tasks  and  TI  materials  were 
compared.  For  troubleshooting  there  were  two  types  of  TI  material:  NTIPS  electronically 
displayed  FIND  and  conventional  Technical  Manual  (Conv(P)).  For  corrective  maintenance 
the  TI  materials  were  NTIPS  electronically  displayed  (NTIPS  (E)),  NTIPS  on  paper  (NTIPS 
(P)),  and  conventional  Technical  Manual  (Conv(P)).  Appendix  C  provides  examples  of  each 
type  of  TI.  Table  3  shows  the  plan  for  assigning  24  subjects  (12  experienced  and  12 
inexperienced)  to  combinations  of  TI  types  and  maintenance  tasks.  Experienced  (EXP) 
subjects  were  defined  as  those  technicians  who  had  more  than  one  year  of  experience  on 
maintenance  of  radars  or  related  electronic  equipment;  inexperienced  subjects  (INEXP) 
had  less  than  this.  All  subjects  were  Electronic  Technicians  (ETs). 

According  to  the  Test  Plan  half  of  the  12  subjects  in  each  experience  group  would 
perform  troubleshooting  using  FIND  for  Fault  1  and  the  conventional  Technical  Manual  for 
Fault  2;  the  remaining  six  subjects  would  perform  troubleshooting  on  Fault  1  with  the 
conventional  Technical  Manual  and  on  Fault  2  with  FIND.  Fault  1  was  always  performed 
first.  For  corrective  maintenance,  each  of  the  three  TI  types  was  paired  with  each  of 
three  different  task  sets:  (1)  remove/install  variable  delay  line,  (2)  remove/install  circuit 
cards,  and  (3)  remove/install  support  brackets.  Assignment  of  subjects  to  these 
combinations  is  shown  in  Table  3.  Subjects  used  the  same  TI  for  installation  that  they 
used  for  removal  (e.g.,  SI  uses  NTIPS(E)  to  remove  the  variable  delay  line  and  then  later 
to  reinstall  the  delay  line). 

The  test  design  as  shown  in  Table  3  is  reproduced  from  the  Test  Plan.  As  the  Test 
progressed,  some  modifications  in  the  Test  procedure  were  made  based  on  the  availability 
of  experienced  subjects.  In  the  actual  Test,  11  experienced  and  13  inexperienced  subjects 
were  used.  Thus  the  Test  included  the  following  assignment  of  subjects  to  TI  conditions 
and  orders: 

o  Troubleshooting  Fault  1: 

-  FIND:  6  experienced;  6  inexperienced 

-  Conv(P):  5  experienced;  7  inexperienced 
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Table  2.  Identification  and  definition  of  troubleshooting  and 
corrective-maintenance  tasks. 


Task  Title 


1.  Ready  AN/SPA-25D  for 
Troubleshooting  and 
Corrective  Maintenance 


2.  Verify  the  Fault 


3.  Troubleshoot  to  Isolate  Faulty 
Component 


4.  Remove  Faulty  Component 


5.  Install  a  New  Component 


Task  Definitions 


Before  any  maintenance  work  can  be  done, 
certain  safety,  power,  and  system  condi¬ 
tions  must  be  set.  Accomplishment  of  this 
task  establishes  these  conditions. 

The  technician  is  told  that  a  malfunction 
exists  in  the  AN/SPA-25D.  The  technician 
selects  the  relevant  TI  and  follows  its 
instructions  to  verify  that  the  reported 
malfunction  does  in  fact  exist.  (In  this 
test,  an  appropriate  fault  has  actually  been 
inserted  into  an  operational  radar  repeater 
by  engineers  of  the  Naval  Sea  Combat 
Systems  Engineering  Station  to  ensure  a 
realistic  procedure.)  Fault  symptoms 
resulting  from  fault  verification  serve  as 
the  basis  for  entering  the  FIND  automated 
troubleshooting  system  or  the 
troubleshooting  part  of  the  conventional 
Technical  Manual. 

In  this  task  the  technician  follows  the  trou¬ 
bleshooting  instructions  of  his  TI  to 
identify  the  component  causing  the  fault 
symptom;  i.e.,  to  perform  fault  isolation. 
For  the  NTIPS  TI,  these  step-by-step  trou¬ 
bleshooting  instructions  are  called  FIND 
(Fault  Isolation  by  Nodal  Dependency). 

The  technician  obtains  these  instructions 
by  interacting  with  the  NTIPS  electronic 
delivery  device. 

The  technician  begins  the  process  of  cor¬ 
recting  the  malfunction  by  removing  the 
component  which  his  testing  has  identified 
as  faulty. 

After  obtaining  a  working  component  from 
Supply,  the  technician  installs  it  in  the 
system  in  place  of  the  faulty  component  he 
has  removed. 
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Table  2  (Continued) 


Task  Title 


6.  Conduct  Operational  Check 


7.  Restore  the  AN/SPA-25D  to 
Operational  Condition 


8.  Complete  Maintenance 
Records 


Task  Definitions 


The  technician  performs  an  operational 
check  of  the  radar  repeater  to  verify  that 
preceding  actions  (1)  have  eliminated  the 
malfunction  and  (2)  have  not  introduced  a 
new  fault  into  the  system. 


The  technician  restores  the  radar  repeater 
to  operational  readiness  by  eliminating 
conditions  which  were  changed  to  permit 
maintenance. 

The  technician  reports  the  completed  work 
on  the  appropriate  Maintenance  Action 
Report  (2-KILO). 
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o  Troubleshooting  Fault  2: 

-  FIND:  5  experienced;  7  inexperienced 

-  Conv(P):  6  experienced;  6  inexperienced 

o  Corrective  Maintenance:  NTIPS(E);  NTIPS(P);  Conv(P) 

-  4  experienced 

-  4  inexperienced 

o  Corrective  Maintenance:  NTIPS(P);  Conv(P);  NTIPS  (E) 

-  3  experienced 

~  5  inexperienced 

o  Corrective  Maintenance:  Conv(P);  NTIPS(E);  NTIPS(P) 

-  4  experienced 

~  4  inexperienced 

The  average  experience  level  of  experienced  and  inexperienced  subjects  is  shown 
below.  It  can  be  seen  that  experienced  subjects  had  an  average  of  one  and  a  half  years 
experience  with  radar  maintenance  and  a  total  of  almost  seven  years  experience  in  the 
Navy.  The  average  for  the  inexperienced  subjects  was  three  months  for  radars  and 
approximately  three  and  a  half  years  in  the  Navy. 


Test  subjects'  experience  and  duty  station 


Technicians 


Number  in  Each 

Average  Time  Rate 

In  Navy  With  Radar  E4  E5  E6 


Experienced  6  yrs.  11  mos.  1  yr.  6  mos.  6  14 

Inexperienced  3  yrs.  5  mos.  3  mos.  9  3  1 


Duty  Station 


Ships: 

USS  Fairfax  County  (LST-1193) 

USS  Dahlgren  (DDG-43) 

USS  Iwo  Jima  (LPH-2) 

USS  Nashville  (LPD-13) 

Shore  Stations: 

NTCC  -  Navy  Telecommunications  Center 
FTC  ~  Fleet  Training  Center 
COMNAVSURFLANT 
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3.4  PRETEST  EVENT 


During  the  period  29  June-2  July,  1987,  four  ETs  performed  all  maintenance  tasks 
planned  for  the  test.  These  efforts  were  carefully  observed  by  test  personnel  to  provide  a 
basis  for  refining  the  Test  Plan  to  ensure  that  it  was  optimal  for  (1)  efficient  use  of  test 
subjects  and  observer  personnel;  (2)  safe  conduct  of  the  tests;  and  (3)  achieving  the  test 
objectives.  The  results  of  this  pretest  event  were  formally  documented.* 

The  Pretest  event  thus  constituted  a  verification  of  the  experimental  Technical 
Information,  a  vital  process  in  the  generation  of  all  system-related  TI  required  to  ensure 
the  operational  suitability  (consistency  with  fleet  procedures,  technician  capability,  and 
operational  environment)  of  the  TI. 

As  a  result  of  the  Pretest  event  and  additional  trips  to  Norfolk, 

1.  Final  changes  were  made  to  the  experimental  TI  (including  the  addition 
of  card  locator,  test  point  locator,  and  waveform  graphics  to  FIND  and 
the  modification  of  the  step-by-step  test  in  FIND) 

2.  Logistic  arrangements  for  test  conduct  were  finalized 

3.  Final  changes  to  the  test  procedure  were  incorporated  to  increase  test 
efficiency. 


3.5  TEST  SCHEDULE  AND  PROCEDURE 

The  Test  was  conducted  during  the  period  from  September  8  to  September  24,  1987. 
One  to  three  subjects  performed  the  test  tasks  each  day.  Two  AN/SPA-25D  radar 
repeaters  were  made  available  by  the  Naval  Sea  Combat  Systems  Engineering  Station, 
making  it  possible  to  test  two  subjects  simultaneously.  The  schedule  of  events  for  each 
subject  was  as  follows: 

o  Attend  a  test  briefing  presented  by  an  Essex  Corporation  data 
collector.  This  briefing  covered  the  test  purpose,  the  tasks  to  be 
performed,  the  estimated  time  required;  and  the  importance  to  the 
program  of  filling  out  the  preference  questionnaire  and  participation 
in  the  debriefing. 

o  Attend  a  technical  briefing  presented  by  a  Hughes  Aircraft  Company 
representative  or  a  computer  specialist  from  DTRC.  This  briefing 
covered  FIND  and  the  electronically  displayed  and  paper  versions  of 
NTIPS  TI  for  corrective  maintenance.  As  part  of  this  briefing  the 
subject  was  given  hands-on  experience  with  the  TI,  the  display 
device,  the  keyboard,  and  the  touch  panel. 


Naval  Technical  Information  Presentation  System:  Initial  Evaluation  AN/SPA- 
25D  Radar  Repeater,  August,  1987  (Essex  Corporation) 
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o  Perform  fault  verification  and  troubleshooting  using  either  FIND  or 
conventional  paper  T1  for  Fault  1  and  then  using  the  other  type  of  TI 
for  Fault  2. 

o  Fill  out  a  questionnaire  comparing  FIND  with  the  conventional 
Technical  Manual  for  troubleshooting. 

o  Perform  corrective  maintenance  with  each  of  the  following  TI  types: 

NTIPS  electronically  delivered,  NTIPS  on  paper,  and  conventional 
Technical  Manual. 

o  Fill  out  a  questionnaire  designed  to  obtain  evaluative  comments  from 
the  test  subjects  comparing  effectiveness  of  NTIPS  TI  delivered  by 
paper  or  electronically  and  conventional  Technical  Manual  for 
corrective  maintenance. 

o  Participate  in  a  debriefing  interview  to  evaluate  the  various  TI  forms 
and  media  used  and  their  relative  effect  on  the  speed  and  accuracy  of 
task  performance.  Provide  opinions  on  new  TI  approaches. 

3.6  THE  ELECTRONIC  DELIVERY  DEVICE 

The  electronic  delivery  system  used  for  troubleshooting  tests  was  an  AT&T  3B-2/300 
computer  with  an  AT&T  6300  terminal  equipped  with  touch  screen  and  keyboard.  The 
touch  screen  worked  by  recording  a  signal  when  a  finger  interrupted  the  infrared  beams 
which  form  a  matrix  across  the  front  of  the  screen.  When  a  pair  of  beams  is  broken  by  a 
finger  passing  through  the  matrix,  the  corresponding  point  on  the  screen  is  identified  as 
being  "touched."  The  computer  terminal  was  hooked  up  to  a  printer. 

The  electronic  delivery  device  used  for  the  corrective  maintenance  TI  was  a  Zenith 
Z-248.  This  system  was  chosen  because  its  response  time  is  much  faster  than  the  AT&T 
computer.  Both  computers  and  the  printer  were  located  on  a  cart  near  the  radar 
repeater.  To  accommodate  simultaneous  testing  on  two  radar  repeaters,  two  sets  of 
computers  were  also  provided. 

3.7  DATA  COLLECTION  CATEGORIES  AND  PROCEDURES 

Data  collection  categories  included  (1)  two  performance  measures  (performance 
time  and  errors  committed  during  task  performance),  (2)  one  descriptive  measure  (actions 
engaged  in  by  technicians  during  task  performance),  (3)  technicians'  subjective  ratings 
(poor  to  excellent)  of  the  quality  of  the  text  and  graphics  which  made  up  the  TI  they  used, 
and  (4)  technicians'  preferences  for  electronic  or  paper  presentation  of  TI.  Sample  data- 
collection  forms  for  troubleshooting  and  for  corrective-maintenance  task  performance  are 
provided  in  Appendix  A.  The  data  collection  form  provides  space  to  record  the  actions 
engaged  in  by  the  subject  when  performing  each  step  in  the  procedure,  the  time  to 
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perform  each  type  of  action,  and  any  errors  made  by  the  subject.  This  form  lists  the  task 
steps  to  be  performed. 

The  subjects'  actions  during  the  conduct  of  the  test  tasks  were  codified  as  follows: 

o  TI  —  Using  Technical  Information 

o  C  —  Communicating 

o  W  —  Performing  work  on  the  system 

o  TI/C  —  Perusing  TI  and  communicating  at  the  same  time  (back  and 
forth  between  perusing  TI  and  talking) 

o  TI/W  —  Perusing  TI  and  working  (actually  moving  or  changing  parts) 
at  the  same  time  —  back  and  forth  between  TI  and  work 

o  TI/E  —  Perusing  TI  and  examining  equipment  (back  and  forth 
between  TI  and  looking  at  equipment). 

These  categories  were  used  in  monitoring  field  test  performance  to  provide  a  record 
showing  what  the  subject  did  to  complete  each  maintenance  step.  The  procedure  used  was 
to  record  each  action  code  while  the  corresponding  action  was  occurring  and  to  record  the 
elapsed  time  until  the  subject  went  on  to  another  type  of  action.  Timing  was  initiated  at 
the  beginning  of  each  subtask.  Thus,  within  each  subtask  or  step,  it  was  possible  to 
determine  the  amount  of  time  spent  working,  examining  equipment,  perusing  TI,  etc. 

The  errors  made  by  a  subject  were  recorded  as  they  occurred.  Categories  of  errors 
included: 

o  Makes  False  Starts:  Begins  to  work  on  equipment,  stops,  looks  back 
at  TI,  and  then  starts  over. 

o  Looks  in  Wrong  Location:  The  subject  attempts  to  find  a  component 
in  a  different  location  from  the  location  listed  in  the  step 
description. 

o  Works  on  Wrong  Part:  The  subject  works  on  the  wrong  part  or  uses 
the  wrong  tool. 

o  Other  Errors:  Errors  not  covered  by  the  above  defined  categories. 

Measures  of  performance  time  and  accuracy  were  analyzed  by  subtask,  by  type  of  TI 
used,  and  by  subject  experience.  The  outcomes  of  these  analyses  are  reported  in  the 
Results  section  (4.0).  In  addition,  a  post-test  debrief  of  the  test  subjects  was  conducted. 
This  debrief  was  used  to  determine  the  subjects'  attitudes  (e.g.,  acceptance  or  dislike) 
toward  individual  types  of  TI  and  the  presentation  media  used.  By  means  of  a  written 
questionnaire,  subjects  were  asked  to  express  their  reactions,  for  or  against,  specific  TI 
characteristics,  including  style,  content,  format,  delivery  medium,  and,  in  general,  the 
ease  of  use  of  the  TI.  It  is  important  to  assess  which  TI  characteristics  caused  the 
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technicians  to  view  TI  as  beneficial  and  easy  to  use,  or  —  on  the  other  hand  —  trou¬ 
blesome  and  confusing.  A  negative  reaction  in  this  category  of  data  might  indicate  areas 
where  improvement  could  be  required  before  such  a  system  could  be  introduced  into  the 
Navy.  The  questionnaire  used  to  obtain  the  users'  reactions  appears  in  Appendix  B.  This 
questionnaire  includes  a  scale  to  allow  the  respondents  to  report  the  intensity  of  their 
reactions,  negative  or  positive,  on  a  scale  of  1  to  5,  to  the  individual  TI  characteristics. 
Subjects  were  also  asked  to  compare  the  types  of  TI  for  each  task  and  rank  them 
according  to  their  preference. 

In  addition,  each  subject  was  interviewed  to  allow  him  to  express  any  opinions  not 
covered  by  the  questionnaire.  An  outline  used  to  conduct  this  interview  appears  as  the 
last  page  of  Appendix  B. 
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4.0  RESULTS 


4.1  SUMMARY  OF  RESULTS 

The  objectives  of  the  NTIPS  Test  were  as  follows: 

o  Compare  the  performance  of  enlisted  maintenance  technicians  using 
TI  prepared  under  NTIPS  procedures  to  the  performance  of 
technicians  using  conventional  TI  (the  paper  AN/SPA-25D  Technical 
Manual:  NAVSEA  0967-LP-445-8010). 

o  Compare  technicians'  performance  when  using  (NTIPS)  TI  printed  on 
paper  to  performance  using  (NTIPS)  TI  presented  via  an  electronic 
display  device. 

o  Establish  which  design  characteristics  of  NTIPS  TI  are  most  effective 
and  least  effective  in  an  operational  situation. 

o  Assess  user  acceptance  of  the  NTIPS  modes  (medium,  content, 
format,  and  style)  of  TI  presentation. 

In  addition,  the  Test  was  designed  to  provide  an  evaluation  of  the  NTIPS  TI  and  various 
aspects  of  the  delivery  device  in  such  a  way  as  to  indicate  the  need  for  improvements  or 
modification  in  either  the  TI  or  in  the  display  system;  i.e.,  the  Test  was  designed  to 
provide  a  "formative  evaluation."  The  data  on  subject  preference  for  various  aspects  of 
the  NTIPS  TI  and  the  use  of  electronic  delivery  of  TI  have  been  analyzed,  and  the  results 
are  reported  below.  The  general  findings  were  that  a  very  high  percentage  (92%)  of  the 
subjects  preferred  electronic  delivery  of  TI  over  conventional-paper  TI  presentation. 
They  were  able  to  use  NTIPS  TI  to  troubleshoot  more  accurately  and  with  greater  speed 
than  with  the  conventional  Technical  Manual.  For  Corrective  Maintenance,  subjects 
performed  at  the  same  accuracy  and  speed  regardless  of  the  type  of  TI  used.  A  consensus 
of  the  comments  about  NTIPS  is  as  follows: 

o  Electronic  delivery  provides  faster  and  easier  access  to  desired 
sections  of  the  TI;  the  technician  does  not  need  to  look  through  large 
volumes  of  paper.  Access  can  be  achieved  by  one  or  two  keystrokes. 

o  FIND  is  easy  to  follow;  the  integration  of  text  and  graphics  was 
particularly  useful. 

o  FIND  provides  useful  guidance  to  the  inexperienced  technician. 

o  Electronic  TI  saves  space  and  is  easier  to  update  than  conventional 
paper. 

o  The  computer  response  time  for  FIND  is  too  slow;  technicians  may 
spend  several  minutes  waiting  for  the  required  data  to  appear. 

o  The  quality  of  the  Corrective  Maintenance  TI  graphics  in  any  future 
electronic  delivery  system  must  be  improved  before  being 
implemented  in  the  Fleet. 
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A  more  detailed  review  of  these  comments  is  provided  in  the  section  on  Preference  Data 
(4.4). 

4.2  PERFORMANCE  TIME  AND  ACTIONS 

As  noted,  several  classifications  were  established  for  collecting  data  describing 
subjects'  actions  during  the  task  performance.  It  was  observed  that  two  categories,  TI  use 
(TI)  and  work  performance  (W),  accounted  for  most  of  the  performance  time.  Based  on 
this  finding,  times  spent  on  all  other  action  categories  (e.g.,  communicating,  examining 
the  equipment,  etc.)  were  combined  and  labeled  "miscellaneous  (M)"  for  purposes  of 
analysis.  Analyses  of  the  performance  times  were  conducted  both  for  troubleshooting  and 
corrective  maintenance. 

4.2.1  Troubleshooting  Performance  Times 

Two  troubleshooting  tasks  were  performed  by  each  subject  technician.  Half  of  the 
24  subjects  used  FIND  and  half  used  the  conventional  Technical  Manual  in  solving  Fault  1; 
when  working  on  Fault  2  the  types  of  TI  were  reversed.  Thus  each  subject  used  both  FIND 
and  the  conventional  Technical  Manual  when  performing  troubleshooting.  In  the 
conventional  Technical  Manual  (if  the  Fault  Logic  Diagram  is  followed),  three  test  points 
are  required  to  isolate  Fault  1  and  four  test  points  to  isolate  Fault  2;  in  FIND,  four  test 
points  were  selected  for  Fault  1  and  five  test  points  for  Fault  2. 

The  conventional  Technical  Manual  provided  several  types  of  data  to  support 
troubleshooting.  These  data  were  in  a  variety  of  formats  and  were  scattered  throughout 
the  manual,  making  it  necessary  for  the  technicians  to  flip  back  and  forth.  The  types  of 
information  presented  in  the  conventional  Technical  Manual  include 
o  Table  of  Contents 
o  Troubleshooting  index 
o  General  troubleshooting  instructions 
o  Signal  flow  diagrams 
o  Schematics  including  correct  waveforms 
o  Fault  logic  diagram  (includes  fault  verification  3teps) 
o  Graphic  showing  circuit  can'd  locations 
o  Graphics  showing  test  point  locations  for  each  card. 

FIND  presented  an  integrated  set  of  procedural  steps  and  graphics  (e.g.,  card 
locations,  test  point  locations,  correct  waveforms)  for  each  test  point.  FIND  operates  by 
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interpreting  the  fault  symptoms  entered  by  the  subject  to  select  the  optimum  test  point 
to  begin  troubleshooting.  As  the  subject  technician  enters  each  test  result  (as 
satisfactory  or  unsatisfactory),  the  FIND  system  directs  the  technician  to  the  next  most 
logical  test;  this  process  is  repeated  until  the  fault  is  isolated.  The  specific  data 
presented  by  FIND  for  each  recommended  test  are 
o  Control  settings  on  the  repeater 
o  Oscilloscope  settings 
o  Circuit  card  location  and  test  point  location 
o  Procedural  steps 
o  Correct  waveform  graphic. 

Table  4  compares  the  performance  times  for  experienced  subjects  using  NTIPS 
electronically  displayed  TI  (FIND)  with  the  performance  times  of  subjects  using  the 
conventional  Technical  Manual  for  troubleshooting  Faults  1  and  2.  The  results  for  Fault  1 
show  that  the  experienced  technicians  found  the  fault  28%  faster  using  FIND  than  when 
using  the  conventional  Technical  Manual.  For  Fault  2,  the  experienced  technicians 
performance  time  was  16%  faster  with  FIND  as  compared  to  the  conventional  Technical 
Manual.  It  may  be  noted  from  Table  4  that  the  major  portion  of  the  differences  in 
performance  time  are  in  time  spent  using  TI.  Troubleshooting  performance  time  could  be 
further  reduced  by  reducing  the  time  the  FIND  program  requires  to  provide  "next  step” 
data  to  the  technician.  An  analysis  of  the  existing  response  time  shows  that 
approximately  60%  of  the  time  attributed  to  TI  use  is  spent  waiting  for  the  next 
instruction.  This  time-consuming  feature  of  FIND  is  a  function  of  the  current 
programming  of  the  system  and  of  the  speed  of  the  computer  used  to  host  the  program  at 
this  time. 

Table  5  compares  the  performance  times  for  inexperienced  subjects  using  FIND  and 
the  conventional  Technical  Manual  for  troubleshooting  Faults  1  and  2.  Performance  times 
for  these  inexperienced  technicians  are  30%  faster  for  FIND  on  Fault  1  and  22%  faster  for 
FIND  on  Fault  2.  Most  of  this  difference  can  be  attributed  to  the  amount  of  time  spent 
using  the  TI.  This  is  particularly  true  on  Fault  1  where  in  fault  isolation  alone  subjects 
spent  an  average  of  about  8.5  minutes  longer  searching  through  and  interpreting  the  TI  in 
the  conventional  Technical  Manual. 

Table  6  summarizes  the  percentage  of  time  saved  using  FIND  for  all  subjects.  For 
experienced  subjects  the  time  saved  is  22%;  for  inexperienced  subjects  the  time  saved  is 
26%. 
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Table  4.  Troubleshooting  mean  performance  times: 
experienced  subjects. 

(times  in  minutes:seconds) 


FIND 

(6) 

Fault  1 

Tech.  Manual 
(5) 

FIND 

(5) 

Fault  2* 

Tech.  Manual 
(6) 

Verify  Fault 

4:05 

5:00 

3:49 

6:01 

Isolate  Fault 

TI 

11:27 

20:00 

13:34 

17:30 

Work 

7:19 

8:01 

7:47 

7:23 

Misc 

1:00 

1:57 

1:11 

Total 

23:51 

33:01 

27:07 

32:05 

♦Excludes  data  for  two  instructors  who  taught  this  problem  at  the  Fleet 
Training  Center.  The  total  time  including  these  subjects  is  24:31. 
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Table  5.  Troubleshooting  mean  performance  times: 
inexperienced  subjects. 

(times  in  minutes:seconds) 


Fault  1 


Fault  2 


FIND 

(6) 

Tech.  Manual 
(7) 

FIND 

(7) 

Tech.  Manual 
(6) 

Verify  Fault 

3:41 

8:00 

2:47 

7:44 

Isolate  Fault 

TI 

11:32 

20:49 

13:34 

16:35 

Work 

8:27 

6:24 

7:06 

5:26 

Misc 

1:00 

1:09 

1:48 

Total 

24:40 

35:13 

24:36 

31:33 

Table  6.  Troubleshooting  percentage  savings  using  FIND 
all  faults  and  all  subjects. 


Subjects 

Fault  1 

Fault  2 

Total 

Percent  Savings 
Using  FIND 

Experienced 

27.8% 

15.5% 

21.6% 

Inexperienced 

29.9% 

22.0% 

26.0% 

Mean  Total 

28.8% 

18.8% 

23.8% 
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4.2.2  Corrective  Maintenance  Performance  Times 


Three  corrective  maintenance  tasks  were  used;  each  required  seven  work  steps  for 
removal  and  seven  work  steps  for  replacement.  The  tasks  consisted  of  the  following 
actions: 

o  Remove/install  Variable  Delay  Line 
o  Remove/install  Circuit  Card  Assemblies 
o  Remove/install  Support  Brackets 

Each  subject  performed  all  three  of  these  tasks,  using  a  sequence  of  TI  types  as 
shown  in  Table  3.  Data  on  overall  performance  times  and  time  spent  in  each  activity 
were  recorded  for  each  corrective  maintenance  task/TI  combination.  A  review  of  these 
data  led  to  the  decision  to  present  only  those  performance  time  data  describing  the 
removal  of  parts.  It  was  observed  that  many  of  the  subjects  relied  on  their  memories  of 
the  removal  task  and  did  not  use  the  TI  to  perform  the  reinstallation.  As  a  result,  it  was 
decided  that  the  installation  tasks  did  not  represent  a  valid  test  of  the  TI. 

Table  7  shows  the  corrective  maintenance  mean  performance  times  for  both 

experienced  and  inexperienced  subjects.  The  following  points  can  be  made  with  regard  to 

experienced  subjects'  performance  time  for  each  type  of  TI: 

o  Time  spent  in  using  the  TI  was  the  longest  for  the  conventional 
Technical  Manual  (Conv  (P))  in  all  three  tasks. 

o  The  best  performance  times  for  removal  of  the  Variable  Delay  Line 
and  the  Circuit  Card  Assemblies  were  obtained  when  using  the  NTIPS 
paper  presentation;  the  conventional  Technical  Manual  required  the 
greatest  amount  of  time  for  the  Variable  Delay  Line  while  the  NTIPS 
electronic  delivery  required  the  most  time  for  removal  of  the  Circuit 
Cards. 

o  There  were  no  significant  differences  in  time  spent  using  TI  or  in 
total  performance  time  for  removal  of  the  Support  Brackets. 

The  time  spent  by  inexperienced  subjects  in  using  the  different  types  of  TI  (first 
data  line)  indicates  that  the  conventional  Technical  Manual  (Conv  (P))  required  the  most 
time  for  Circuit  Card  and  Support  Bracket  removal  while  NTIPS  electronically-delivered 
TI  (NTIPS  (E))  required  the  most  time  for  removal  of  the  Variable  Delay  Line.  Within 
each  task  the  overall  performance  times  for  inexperienced  subjects  were  similar  for  all  TI 
types  for  removal  of  Support  Brackets  and  the  Variable  Delay  Line.  For  the  removal  of 
Circuit  Cards,  using  the  conventional  Technical  Manual  required  2.65  to  4.3  minutes 
longer  than  the  two  NTIPS  presentations. 

These  results  suggest  that  NTIPS  TI  is  as  effective  as  the  conventional  Technical 
Manual  when  evaluated  in  terms  of  corrective  maintenance  performance  time.  As  will  be 


Table  7.  Corrective-maintenance  mean  performance  times, 
(times  in  minutes:seconds) 

Experienced  subjects 


Activity  Variable  Delay  Line  Circuit  Card  Support  Brackets 

Assemblies 


* 

NTIPS 

(E/4) 

NTIPS 

(P/3) 

TM 

(4) 

NTIPS 

(E/4) 

NTIPS 

(P/3) 

TM 

(4) 

NTIPS 

(E/4) 

NTIPS 

(P/3) 

TM 

(4) 

TI 

3:27 

2:52 

5:37 

5:54 

4:18 

7:12 

4:46 

5:09 

5:28 

Work 

6:12 

5:10 

6:48 

7:42 

5:57 

5:58 

8:20 

9:11 

6:48 

Misc 

2:35 

2:43 

2:55 

2:50 

1:23 

:08 

1:38 

:58 

2:58 

Total 

12:14 

10:45 

15:20 

16:26 

11:38 

13:18 

14:44 

15:18 

15:14 

Inexperienced  subjects 

Activity 

Variable  Delay  Line 

Circuit  Card 
Assemblies 

Support  Brackets 

* 

NTIPS 

(E/4) 

NTIPS 

(P/5) 

TM 

(4) 

NTIPS 

(E/4) 

NTIPS 

(P/5) 

TM 

(4) 

NTIPS 

(E/4) 

NTIPS 

(P/5) 

TM 

(4) 

TI 

5:05 

3:06 

3:49 

5:28 

3:50 

7:19 

3:44 

4:15 

5:09 

Work 

4:25 

7:24 

7:50 

6:23 

6:24 

8:16 

8:51 

9:44 

6:25 

Misc 

2:38 

1:55 

1:02 

1:30 

1:27 

:25 

1:20 

1:48 

3:11 

Total 

12:08 

12:25 

12:41 

13:21 

11:41 

16:00 

13:55 

15:47 

14:45 

♦Letters  and  numbers  in  parentheses  indicate  medium  of  T1  delivery  and 
number  of  subjects,  respectively. 
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seen  in  the  discussion  of  preferences,  technicians  criticized  the  graphics  in  both  the  paper 
and  electronic  versions  of  NTIPS.  The  graphics  problems  are  correctable  and  with 
improved  graphics,  the  technicians'  performance  when  supported  by  NTIPS  should  also 
improve. 

4.3  PERFORMANCE  ACCURACY 

Data  collectors  observed  and  recorded  two  types  of  accuracy  problems  during  the 
technicians'  maintenance  performanc 

a.  Inaccuracies  corrected  by  minor  prompts 

o  Performance  errors  of  omission  and  commission,  e.g.,  performed  a 
step  out  of  sequence. 

o  Difficulties  in  interpreting  TI  or  matching  TI  to  equipment,  e.g., 
interpreting  a  waveform. 

b.  Inaccuracies  corrected  by  significant  assistance 

o  Extensive  inactivity  or  work  which  was  irrelevant  to  the  task,  e.g.,  15 
minutes  of  no  work  relevant  to  the  task. 

Both  types  of  inaccuracy  were  observed  during  troubleshooting  performance.  Only 
the  minor  prompt  type  of  performance  inaccuracy  was  noted  during  corrective 
maintenance.  Separate  accuracy  analysis  results  are  described  below  for  troubleshooting 
and  corrective  maintenance. 

4.3.1  Troubleshooting  Accuracy 

a.  Inaccuracies  Requiring  Significant  Assistance.  Subjects  were  given  assistance 
if  after  15  minutes  of  work  they  had  not  yet  performed  any  checks  or  tests  relevant  to  the 
assigned  troubleshooting  problem.  The  judgment  as  to  whether  significant  assistance  was 
needed  was  made  by  Test  personnel  and  was  based  on  the  optimum  troubleshooting 
sequences  included  in  FIND  and  in  the  conventional  Technical  Manual.  Table  8  presents 
the  number  of  signficiant  assists  needed  by  the  technicians  using  FIND  and  the 
conventional  Technical  Manual  to  solve  the  troubleshooting  problems.  None  of  the 
technicians  using  FIND  required  signficant  assistance  to  solve  their  troubleshooting 
problems.  Four  of  the  experienced  technicians  (36%)  and  six  of  the  inexperienced 
technicians  (46%)  required  significant  assistance  to  solve  their  troubleshooting  problem 
when  using  the  conventional  Technical  Manual. 

b.  Minor  Inaccuracies  Requiring  Prompts.  Table  9  shows  the  results  of  the 
analysis  of  this  class  of  performance  inaccuracies  for  troubleshooting.  For  the 
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Table  8.  Signf leant  assistance  needed  in  fault  isolation. 


FIND 

TM 

Solved 

Needed 

Solved 

Needed 

Problem 
without  Help 

Help 

Problem 
without  Help 

Help 

Experienced 

11 

0 

7 

4 

Inexperienced 

13 

0 

7 

6 

100%  0%  58%  42% 


Table  9.  Troubleshooting, 
(number  of  minor  prompts) 


Subjects 

FIND 

Tech.  Manual 

Fault  1 

Experienced 

1 

8 

Sweep  Resolver 

Inexperienced 

0 

21 

Total  Fault  1 

1 

29 

Fault  2 

Experienced 

3 

7 

Main  Gate  Generator 

Inexperienced 

3 

3 

Total  Fault  2 

6 

10 

Total 

Both  Faults 

7 

39 
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technicians  using  FIND,  six  of  the  seven  inaccuracies  occurred  during  the  work  on  Fault  2. 
Four  of  these  prompts  involved  a  problem  in  locating  a  circuit  card.  An  analysis  of  the 
work  site  showed  that  this  inaccuracy  occurred  when  the  graphic  display  on  the  computer 
was  positioned  so  that  the  technicians  had  to  turn  180  degrees  from  the  repeater  to  see 
the  computer  screen.  After  viewing  the  screen  and  returning  their  attention  to  the 
repeater,  they  expected  to  see  a  mirror  image.  As  a  result  they  began  working  on  the 
circuit  card  on  the  wrong  side  of  the  repeater.  This  result  has  implications  for  the 
placement  of  the  computer  at  the  worksite. 

Table  9  shows,  for  the  technicians  using  the  conventional  Technical  Manual,  a  total 
of  39  minor  prompts  to  overcome  inaccuracies;  29  of  the  prompts  were  given  during  the 
Fault  1  problem  and  10  were  given  during  Fault  2.  Table  10  subdivides  the  conventional 
Technical  Manual  inaccuracies  by  type.  Three  types  accounted  for  87%  of  these 
performance  inaccuracies;  these  types  were  (1)  interpretation  of  the  Fault  Logic  Diagram 
(28%),  e.g.,  the  meaning  of  a  question,  which  path  to  follow  for  a  particular  test  result; 
(2)  finding  data  in  the  conventional  Technical  Manual  (31%),  e.g.,  finding  locator  diagrams 
for  cards  and  test  points;  and  (3)  locating  cards  and  test  points  in  the  repeater  (28%). 

To  help  explain  the  higher  number  of  inaccuracies  in  the  conventional  Technical 
Manual,  Table  11  lists  a  generic  version  of  the  troubleshooting  sequences  for  FIND  and  for 
the  conventional  Technical  Manual.  The  relative  ease  of  using  FIND  is  highlighted  by  the 
fact  that  steps  2  through  6  in  FIND  are  displayed  on  three  separate  frames  which  the 
technician  can  link  by  making  a  single  screen  touch  or  keystroke.  On  the  other  hand,  the 
TI  to  support  the  equivalent  steps  in  the  conventional  Technical  Manual  is  contained  on  6 
to  7  pages  (some  of  which  are  foldouts),  and  the  technician  must  search  for  these  pages 
because  they  appear  in  different  sections  of  the  manual. 

4.3.2  Corrective  Maintenance  Accuracy 

a.  Inaccuracies  Requiring  Significant  Assistance.  No  inaccuracies  of  this  type 
were  recorded  during  corrective  maintenance. 

b.  Minor  Inaccuracies  Overcome  by  Prompts.  Table  12  shows  the  number  of 

minor  prompts  given  to  the  technicians  during  corrective  maintenance.  The  minor 

inaccuracies  requiring  the  prompts  were  of  the  following  types: 

o  Could  not  locate  a  part  after  reviewing  the  graphics  and  the 
procedural  steps  (False  Starts) 

o  Omitted  a  step 

o  Worked  on  the  wrong  part 
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Table  10.  Troubleshooting:  types  of  minor  prompts  needed  for  technicians 
using  the  conventional  Technical  Manual  and  FIND. 


Prompts 

Technical 

Manual 

FIND 

1. 

Find  Fault  Logic  Diagram 

1 

2. 

Interpret  Fault  Logic  Diagram 

11 

3. 

Find  Data  in  Technical  Manual 

12 

4. 

Interpret  Schematics 

4 

5. 

Find  Cards/Test  Points  in  Equipment 

11 

_7 

39 

7 
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Table  11.  Troubleshooting  activity  sequence, 


FIND 

1.  Respond  to  Fault  Verification 
Questions 

2.  Read  TI  on  1st  test  point 
selected  by  FIND 

3.  Read  oscilloscope  set  up 
procedures 

4.  Read  indicator  settings/make 
adjustments 

5.  Read  card  location  diagram 


6.  Read  procedures,  test  point 
location  diagram,  and  correct 
waveform  graphic 

7.  Perform  1st  test 

Repeat  steps  2~7  for  each  test 


Technical  Manual 
1.  Read  Table  of  Contents 


2.  Read  troubleshooting  index 


3.  Locate  and  read  appropriate  signal  flow 
diagram  or  Fault  Logic  Diagram 

4.  Locate  and  review  schematic  for  test 
point  and  waveform 

5.  Locate  and  review  card  location 
graphic 

6.  Locate  and  review  appropriate  test 
point  location  graphic 


7.  Perform  test 

8.  Locate  and  read  signal  flow  diagram  or 
schematic  for  correct  waveforms 

9.  Locate  and  read  appropriate  signal  flow 
diagram  or  the  Fault  Logic  Diagram  for 
the  next  test 

Repeat  steps  4-9  for  each  test 
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Table  12.  Corrective  maintenance, 
types  of  minor  errors. 


NTIPS  (E) 


Error  Category  Exp  Inexp 


1.  False  starts  10  7 

2 .  Worked  on  wrong 

part  3  2 

3.  Omitted  step  —  1 

4.  Performed  steps  in 

wrong  sequence  1 

5 .  Installed  part 

incorrectly  1  2 

TOTALS  15  12 


TI  Type 

NTIPS  (P)  Conv  (P) 

Exp  Inexp  Exp  Inexp 

4  7  5  5 

3—3 
113  4 

1115 

3  112 

9  13  10  19 
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o  Performed  steps  in  the  wrong  sequence 
o  Installed  a  part  incorrectly. 

Overall,  technicans  committed  78  minor  errors.  Thirty-four  were  made  by 
experienced  technicians  and  44  were  made  by  inexperienced  technicians.  Location 
difficulties  accounted  for  the  most  errors  (rows  1  and  2  accounted  for  63%  of  the  total). 
Comparisons  of  the  total  number  of  errors  for  each  experience  group  and  each  type  of  TI 
show  that 

o  Experienced  technicians  made  more  errors  when  using  NTIPS 
electronically  delivered  TI  than  when  using  the  other  two  types  of  TI 

o  Inexperienced  technicians  made  the  largest  number  of  errors  when 
using  the  conventional  Technical  Manual. 

A  frequent  comment  made  by  many  technicians  throughout  the  corrective 
maintenance  tasks  referred  to  the  lack  of  clarity  in  the  NTIPS  graphics  and  the  difficulty 
in  using  these  graphics  to  make  accurate  identifications  of  parts  in  the  equipment.  This 
point  will  be  discussed  in  more  detail  in  the  section  on  preference  data  (4.4). 

4.4  PREFERENCE  DATA  AND  TI  EVALUATION 

A  three-part  preference  questionnaire  (Appendix  B)  was  completed  by  all  subjects. 
The  first  part  of  the  questionnaire  asked  subjects  to  indicate  their  preference  for  FIND  or 
for  the  conventional  Technical  Manual  and  to  rate  the  features  of  the  electronic  delivery 
system  on  a  scale  from  1  to  5,  with  1  being  poor  and  5  being  excellent.  Subjects  were 
scheduled  to  respond  to  this  part  immediately  following  their  troubleshooting  task 
performance.  The  second  part  of  the  questionnaire  asked  subjects  to  rank  the  three  types 
of  TI  used  for  corrective  maintenance  from  most  preferred  to  least  preferred.  In 
addition,  subjects  rated  features  of  the  electronic  delivery  of  corrective  maintenance  TI 
on  a  scale  from  1  to  5.  They  were  provided  with  these  questions  following  the  corrective 
maintenance  task  performance.  At  the  same  time  subjects  were  provided  with  the  third 
part  of  the  questionnaire  which  requested  an  overall  evaluation  and  comments. 

Following  completion  of  the  questionnaire,  subjects  were  debriefed.  During  the 
debriefing  the  following  questions  were  posed  to  each  test  subject: 

1.  If  you  had  a  choice  of  using  TI  electronically  delivered  or  TI  on  paper  to 
perform  troubleshooting  and  corrective  maintenance  tasks,  which  would 
you  choose? 

2.  In  working  with  technical  documentation,  was  it  easier  with  the  TI  elec¬ 
tronically  delivered  or  with  TI  on  paper? 

3.  Which  mode  of  presentation  was  better  organized  for  your  purposes  — TI 
electronically  delivered  or  TI  on  paper? 
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4.  What  do  you  see  as  the  major  advantages  of  the  electronic  presentation? 

Of  the  paper  presentation? 

Table  13  summarizes  the  responses  to  these  questions,  broken  down  by  experience 
level  of  the  technician.  These  results  show  that  92%  of  all  technicians  preferred 
electronic  to  paper  delivery  of  TI  based  on  their  experiences  during  the  Test. 
Additionally,  20  of  the  24  subjects  indicated  that  the  TI  electronically  delivered  was 
easier  to  use  and  better  organized  than  the  technical  manual.  The  technicians  identified 
three  major  advantages  cf  TI  electronically  delivered: 
o  Provides  faster  access 
o  Eliminates  clutter  and  page  flipping 
o  Provides  guidance  for  inexperienced  technicians 

The  major  advantages  of  the  paper  presentation  over  the  electronic  delivery 
were  identified  as  follows: 

o  Schematics  are  provided  in  the  conventional  Technical  Manual  and 
not  in  the  current  electronic  delivery  system.  Several  technicians, 
particularly  those  with  experience,  preferred  to  use  schematics  for 
troubleshooting  rather  than  a  procedural  approach. 

o  The  graphics  for  corrective  maintenance  were  clearer  in  the  paper 
presentation  than  in  the  electronic  presentation. 


Table  13.  Subjects  preference  for  TI  medium. 


Experienced  Inexperienced  Total 


Paper 

Electronic 

Paper 

Electronic 

Paper 

Electronic 

Overall  Preference 

1 

10 

1 

12 

2 

22 

Easier  to  Use 

2 

9 

2 

11 

4 

20 

Better  Organized 

2 

9 

2 

11 

4 

20 

The  subjects’  responses  to  the  preference  items  in  the  questionnaire  are  provided  in 
Tables  14  and  15.  Some  general  findings  regarding  preference  are 

o  For  troubleshooting:  FIND  was  preferred  to  the  conventional  Techni¬ 
cal  Manual  by  all  subjects  for  fault  isolation,  and  by  22  of  the  24 
subjects  for  fault  verification  (Table  14). 

o  For  corrective  maintenance:  For  step  instructions  NTIPS  electronic 
was  preferred  by  63%  of  the  subjects;  17%  preferred  NTIPS  on  paper 
and  20%  preferred  conventional  Technical  Manual  (Table  1 5). 
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o  For  corrective  maintenar  le:  For  graphics,  paper  presentation  either 
NTIPS  or  the  conventional  Technical  Manual  was  preferred  by  75%  of 
the  subjects  (Table  15). 


Table  14.  Troubleshooting:  TI  preferences, 
(number  of  subjects) 


Fault  Verification  Fault  Isolation 

Technical  FIND  Technical  FIND 


Subjects 

Manual 

Manual 

Experienced 

2 

9 

— 

11 

Inexperienced 

— 

13 

— 

13 

Total 

2 

22 

24 

Table  15.  Corrective  maintenance: 

TI  preferences. 

(number  of  subjects) 

Step  Instructions 

Graphics 

NTIPS 

NTIPS  Technical 

NTIPS 

NTIPS 

Technical 

Subjects 

(E) 

(P)  Manual 

(E) 

(P) 

Manual 

Experienced 

7 

2  2 

3 

5 

3 

Inexperienced 

8 

2  3 

3 

4 

6 

Total 

15 

4  5 

6 

9 

9 

In  addition  to  indications  of  preference  for  the  various  types  of  TI  used  in  the  test, 
subjects  were  asked  to  rate  the  information  control,  the  keyboard,  and  the  display  screen 
of  the  electronic  delivery  system.  The  information  control  features  include  menus  and 
procedures  for  locating  information  in  the  data  base;  the  keyboard  features  are  layout  and 
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ease  of  use;  the  screen  features  are  brightness,  glare,  and  resolution.  Table  16  presents 
the  ratings  for  these  features.  The  results  show  that  all  features  were  rated  as  good  (the 
scale  was  1-5,  with  1  being  poor  and  4  being  good). 


Table  16.  Ratings  for  electronic  system  features, 
(scale  1  poor  -  5  excellent) 


Subjects  Information  Control  Keyboard  Screen 


FIND 

NTIPS  (E) 
(Corrective  Main.) 

Experienced 

4.0 

3.7 

4.0 

4.0 

Inexperienced 

4.0 

4.1 

4.1 

4.2 

At  the  end  of  the  questionnaire  sections  dealing  with  troubleshooting  and  corrective 
maintenance,  subjects  selected  from  a  list  of  features  of  the  electronically  presented  TI 
those  features  they  liked  the  most  and  those  they  liked  the  least.  The  results  are  shown  in 
Table  17.  The  same  three  features  were  selected  by  the  test  subjects  as  most  desirable 
for  the  electronically  delivered  troubleshooting  TI  and  for  the  electronically  delivered 
corrective  maintenance  TI,  These  three  features  are 

o  The  step-by-step  instructions  (clarity  of  text  and  procedure) 
o  The  organization  of  the  procedure 
o  The  relationship  of  text  to  graphics  on  each  screen. 

The  features  the  subjects  liked  the  least  differed  for  troubleshooting  and  corrective 
maintenance.  As  mentioned  earlier,  one  of  the  biggest  problems  with  FIND  was  the  slow 
system  response  time;  approximately  60%  of  the  time  spent  using  the  FIND  TI  involved 
waiting  for  the  system  to  process  a  result  or  a  response  and  present  the  next  screen  of 
information.  The  second  major  problem  with  FIND,  as  it  currently  exists,  is  the  inability 
to  back-up  sequentially  or  to  skip  over  information  that  is  already  known  by  the 
technician.  Both  response  time  and  flexibility  of  procedure  are  being  addressed  by  the 
NTIPS  staff,  and  appropriate  modifications  will  be  introduced. 
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Table  17.  Features  of  NTIPS  TI  (electronically  presented) 
selected  as  most  and  least  liked. 


Most  Liked 


Least  Liked 


FIND 


1.  Step  by  step 
instructions 

2.  Organization 

3.  Relationship  of  text 
to  graphics 


1.  System  response  time 

2.  Movement  around  data 
base 


NTIPS  Electronic 
(Corrective  Maintenance) 

1.  Step-by-step 
instructions 

2.  Organization 

3.  Relationship  of  text 
to  graphics 


1.  Size  of  graphics 

2.  Detail  of  graphics 
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For  corrective  maintenance,  the  major  problems  with  the  electronic  TI  involved  the 
graphic  presentations.  Graphics  were  too  small,  too  hard  to  read,  and  in  many  cases,  not 
oriented  properly.  In  addition,  callouts  were  not  always  clear.  Suggestions  made  during 
the  test  include 

o  Increase  graphic  size 
o  Improve  graphic  resolution 

o  Provide  technician  with  a  reference  point  on  the  graphic  that  can  be 
seen  easily  on  the  equipment 

o  Highlight  callouts  for  each  step;  have  technician  indicate  the  step 
and  then  show  relevant  callouts  in  a  brighter  or  different  color. 

o  Provide  less  text  in  specific  text-graphic  modules.  This  procedure 
would  result  in  fewer  callouts  per  screen  and  allow  graphics  to  be 
larger. 

4.5  FLEET  TECHNICIANS'  SUGGESTIONS  ON  HOW  TO  IMPROVE  DEVELOPMENTAL 
NTIPSTI 

The  fleet  technicians  participating  in  the  test  were  asked  during  a  post-test 
interview  what  the  Navy  might  do  to  further  improve  the  electronic  devices  used  to 
deliver  maintenance  information.  A  summary  of  their  suggestions  is  provided  below. 

A.  Usability  of  Graphics  for  Both  the  Paper  and  Electronically  Delivered  Corrective 
Maintenance  Packages 

Virtually  all  participating  technicians  stated  that  the  graphics  for  NTIPS  corrective 
maintenance  TI  were  difficult  to  use.  The  technicians  believed  that  key  equipment-part 
illustrations  were  too  small  and  contained  too  little  detail.  Paper  raphics  were  only 
slightly  better  than  graphics  delivered  electronically.  All  parts  of  the  NTIPS  TI,  including 
graphics  for  corrective  maintenance,  were  authored  once  at  an  NTIPS  authoring  station, 
and  then  mastered  for  paper  and  electronic  delivery;  thus  the  two  types  of  experimental 
TI  used  essentially  the  same  graphics. 

B.  Response  Time  for  NTIPS  Troubleshooting  TI 

The  NTIPS  troubleshooting  TI  (FIND)  is  designed  to  be  interactive  and  as  a  result  is 
presented  only  electronically.  Also,  testing  sequences  for  a  given  symptom  may  not  be 
constant  over  time,  e.g.,  as  the  system  matures,  new  test  times  and  component  failure 
rates  may  be  input  to  FIND  and  the  optimum  testing  sequence  for  a  given 
failure/symptom  may  change.  The  optimum  testing  sequence  is  determined  by  an 
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algorithm  whose  time  to  run  is  excessive,  especially  in  the  AT&T  6300  (a  result  that 
occurred  when  FIND  was  rehosted  to  the  AT&T  equipment  from  the  MODCOMP  used  in 
the  original  development  of  FIND). 

As  a  result  of  this  situation,  technicians  complained  about  the  slow  system  response 
time  for  NTIPS  troubleshooting  TI,  especially  for  the  Fault  Isolation  part  (FIND). 
However,  as  shown  in  Table  6,  even  with  this  slow  response,  FIND  fault  isolation  time 
averaged  24%  faster  than  when  using  the  conventional  Technical  Manual.  Timing  the  wait 
and  TI  display  cycles  in  FIND  showed  that  approximately  60%  of  the  time  spent  using  the 
TI  involved  waiting  for  the  machine  to  respond  with  the  next  TI  frame. 

Improving  this  software  so  that  the  waiting  time  is  reduced  to  half  of  what  it  was 
during  the  test  will  improve  FIND's  time  savings  even  further.  Projected  percentage  of 
time  saved  using  FIND  is  as  follows: 

Total 

Subjects  Fault  1  Fault  2  Savings 

Experienced  39.90%  40.35%  40.13% 

Inexperienced  28.04%  38.60%  33.32% 

Total  33.97%  39.47%  36.73% 

Thus  the  percentage  of  time  saved  by  use  of  FIND  would  increase  to  37%. 

C.  Repetition  in  FIND 

In  the  Fault  Isolation  part  of  FIND  each  test  begins  with  instructions  for  setting  up 
the  test  equipment  and  the  system  hardware.  Invariably,  these  set-ups  are  the  same  for 
successive  tests  even  though,  theoretically,  they  could  differ.  In  the  eyes  of  the 
technician,  the  repetition  is  a  waste  of  time  and  FIND  software  should  be  modified  to 
keep  track  of  and  delete  this  repetition  when  appropriate. 

D.  Use  of  Schematics  to  Support  Troubleshooting 

There  are  two  categories  of  aids  to  support  troubleshooting: 

o  proceduralized  (step-by-step  instructions  which  prescribe  the 
required  steps  in  the  fault  isolation  procedure  ) 
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o  decision  making  (technician  selects  the  checks  and  tests  to  perform 
based  on  his  interpretation  of  the  hardware  dependencies  shown  on  a 
schematic). 

FIND  uses  the  proceduralized  type  of  aid,  but  a  relatively  large  percentage  of  the 
technicians  stated  that  they  would  like  to  see  schematics  also  become  a  part  of  the  NTIPS 
troubleshooting  TI.  This  recommendation  is  consistent  with  other  research  results  which 
indicate  that,  depending  on  the  level  of  experience,  technicians  desire  to  have  a 
schematic  to  use  exclusively  or  to  use  as  an  adjunct  to  proceduralized  instructions.  FIND 
in  its  present  form  includes  very  small  parts  of  a  schematic  diagram.  Generally,  these 
parts  correspond  to  circuit  cards  and  the  partial  schematic  shows  inputs  to  and  outputs 
from  the  circuit  cards. 

E.  Animation  of  Waveform  Portrayal 

Past  research  indicates  that  technicians  have  had  difficulty  using  some  types  of  test 
equipment  and  interpreting  the  test  equipment  outputs.  In  using  FIND,  technicians 
performed  well  on  the  set-up  of  the  oscilloscope,  but  experienced  some  difficulty  in 
assessing  waveforms  as  being  good  or  bad.  One  of  the  technicians  participating  in  the  test 
observed  that  the  TI  showed  the  expected  waveform  as  a  static  picture  but  that  many  of 
the  waveforms  showed  motion  when  seen  on  the  oscilloscope.  He  suggested  that  the 
electronic  display  might  include  animation  to  help  the  user  in  his  assessment  of  these 
types  of  waveforms. 

F.  User  Initiated  Interaction  for  Troubleshooting 

Some  users  suggested  that  they  be  allowed  more  flexibility  in  interacting  with  the  TI 
rather  than  being  told  what  to  do  by  the  machine.  Past  research  has  frequently  produced 
this  user  complaint  about  proceduralized  instructions  and  about  any  automation.  This 
input  is  related  to  the  users'  suggestion  to  include  schematics  as  part  of  the 
troubleshooting  information  (see  item  D  above). 
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5.0  CONCLUSIONS 


5.1  INTRODUCTION 

The  NTIPS  test  and  evaluation  was  conducted  in  two  stages: 

1.  The  preparation  of  test  TI  in  accordance  with  NTIPS  specifications  and 
by  use  of  an  NTIPS-developed  automated  authoring  system. 

2.  The  comparison  of  technician  performance  quality  resulting  from  the  use 
of  NTIPS  and  conventional  TI  in  carrying  out  troubleshooting  and 
corrective-maintenance  tasks  on  an  AN/SPA-25D  radar  repeater. 

Conclusions  drawn  from  the  data  collected  during  both  stages  are  presented  in  the 
following  sections. 

5.2  SUMMARY  OF  CONCLUSIONS 

Results  of  the  NTIPS  field  test  have  shown  that  TI  constructed  by  automated 
authoring  and  according  to  NTIPS  specifications,  when  applied  by  fleet  technicians  in 
operational  maintenance  tasks,  can  significantly  improve  performance  (particularly  trou¬ 
bleshooting);  and  that  the  electronic  presentation  of  maintenance  TI  is  considered  superior 
to  paper  presentation  of  TI  by  92  percent  of  experienced  and  inexperienced  technicians. 

The  tests  also  provided  valuable  guidance,  both  in  the  area  of  specifying  the  most 
effective  TI  (e.g.,  the  need  for  better  graphics)  and  in  the  area  of  electronic-presentation 
approaches  (e.g.,  system  response  time).  A  number  of  these  suggestions  have  already  been 
incorporated  into  NTIPS  approaches;  others  will  form  the  basis  for  further  development. 

Although  it  is  difficult  to  generalize  from  a  test  involving  a  small  population  of 
technicians  working  in  a  specific  maintenance  area,  it  appears  that  fleet  technicians  will 
welcome  the  automated  generation  and  presentation  of  fault-isolation  and  corrective- 
maintenance  TI.  The  innovations  proposed  by  NTIPS  in  TI  content,  format,  style,  and 
organization,  even  when  presented  on  paper,  are  as  effective  and  in  some  cases  more 
effective  than  conventional  paper  Technical  Manuals. 

5.3  CONCLUSIONS  REGARDING  NTIPS  SPECIFICATIONS  AND  AUTHORING  SYSTEM 

FOR  GENERATING  TI 

In  general,  the  TI  contractor  was  able  to  follow  the  NTIPS  TI  specifications  and  was 
also  able  to  use  the  TI  authoring  system  to  prepare  the  experimental  TI  presenting 
AN/SPA-25D  procedures  used  in  the  field  test.  However,  experience  with  this  Test  has 
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shown  that  use  of  an  automated  authoring  system  to  prepare  Navy  weapon  system  TI  in  no 
way  reduces  the  need  for  a  careful  contractually  mandated  quality  assurance  program. 
Specific  observations  recorded  during  the  preparation  process  are  summarized  below. 

5.3.1  Specifications 

No  problems  were  found  regarding  clarity  of  specifications.  However,  test  data 
showed  that  the  contractor  experienced  consistent  problems  in  carrying  out  some  of  the 
aspects  of  TI  generation  required  by  TI  specifications.  Since  some  of  the  NTIPS 
approaches  to  TI  generation  were  entirely  new  to  the  TM  writers,  this  was  not  an 
unexpected  result. 

5.3.2  Computer-Assisted  Authoring 

During  the  TI  generation  process,  extensive  interaction  was  required  between  the  TI 
contractor  and  the  developer  of  the  authoring  programs  due  to  the  radical  difference 
between  the  automated  approach  and  the  manual  approach  to  TI  generation  with  which 
the  contractor  was  familiar.  A  contractor  was  chosen  who  had  little  or  no  experience 
with  TI  automation  in  order  to  elucidate  problems  of  this  nature.  The  "prompting"  feature 
of  the  NTIPS  automated  authoring  systems  proved  not  to  be  particularly  effective  as 
designed.  The  MODCOMP  computer  used  to  host  the  authoring  routine  has  been  made 
obsolete  by  rapid  technological  progress  in  automated  authoring. 

5.4  Conclusions  and  Observations  Based  on  Data  From  Field  Test 

This  section  summarizes  (1)  observations  made  during  the  operational  field  test 
conducted  at  the  Naval  Sea  Combat  Systems  Engineering  Station  and  (2)  the  conclusions 
drawn  concerning  impact  of  the  NTIPS  experimental  TI  on  maintenance  task  performance. 

Reactions  of  test  subjects  to  the  various  TI  features  were  consistent  with  results 
obtained  in  operational  tests  made  previously  by  the  three  services  on  various  kinds  of  TI 
content,  format,  style,  medium,  and  procedural  organization  (including  those  obtained  in 
the  previous  NTIPS  field  test).*  Special  relevance  of  these  reactions  to  improvement  of 
the  NTIPS  technology  (both  TI  and  presentation  methodology)  are  cited  in  the  following 
sections. 


Test  and  evaluation  of  the  Navy  Technical  Information  Presentation  System  (NTIPS): 
F-14A  Experimental  Technical  Information  Field  Test.  June  1987  (Essex 
Corporation) 
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5.4.1  Different  Approaches  to  TI  Use 

Several  approaches  to  the  use  of  TI  were  observed  among  technicians  during  the 
test.  A  common  approach  involved  a  complete  review  of  the  procedure  by  the  technician 
before  any  steps  were  performed,  instead  of  reading  each  step  once  and  then  performing 
it  before  moving  ahead  in  the  TI.  This  observation  implies  that  possible  benefit  would 
result  from  providing  a  browsing  mode  in  the  electronic  delivery  of  task  instructions;  i.e., 
providing  a  summary  of  a  sequence  of  steps  (for  example,  a  kind  of  annotated  checklist)  as 
a  supplement  to  complete  details  for  one  step  at  a  time. 

5.4.2  Troubleshooting  Task  Performance  Time 

Both  experienced  and  inexperienced  subject-technicians  performed  troubleshooting 
faster  using  FIND  than  when  using  the  conventional  Technical  Manual.  For  experienced 
subjects  the  difference  in  performance  time  was  22  percent;  for  inexperienced  subjects 
the  difference  was  26  percent.  These  differences  can  be  attributed  to  two  factors.  First, 
FIND  integrates  all  the  data  needed  by  a  technician  to  perform  a  test.  The  conventional 
Technical  Manual,  on  the  other  hand,  is  organized  so  that  the  technician  must  search  for 
the  desired  information  and  flip  from  one  section  to  another  in  order  to  review  all  the 
relevant  pieces  of  data.  Thus,  the  information-gathering  process  involved  in  using  the 
conventional  Technical  Manual  is  extremely  time-consuming.  The  second  factor 
influencing  differences  in  performance  time  is  that  FIND  selects  the  test  points  for  the 
technician  while  the  conventional  Technical  Manual  makes  the  technician  responsible  for 
test-point  selection.  Selecting  an  approach  and  tracking  down  the  procedure  involves 
time;  and  if  the  approach  is  incorrect,  then  the  technician  must  begin  again. 

An  additional  point  to  be  made  about  FIND  is  that  there  is  an  opportunity  to  make  it 
even  more  efficient  by  improving  the  system  response  time.  In  the  current  form  of  FIND 
the  technician  spends  60  percent  of  his  TI  use  time  waiting  for  the  system  to  present  the 
next  information  screen  (see  Section  4.5  B). 

5.4.3  Minor  Inaccuracies  in  Troubleshooting 

Both  experienced  and  inexperienced  technicians  required  significantly  more  prompts 
to  overcome  minor  inaccuracies  when  using  the  conventional  Technical  Manual  than  when 
using  FIND  for  troubleshooting.  Specifically,  the  total  technician  population  required 
seven  prompts  with  FIND  and  39  prompts  with  the  conventional  Technical  Manual.  Few 
prompts  were  required  with  FIND  because  FIND  provides  all  the  necessary  information  to 
conduct  the  tests.  The  prompts  required  when  the  conventional  Technical  Manual  was 
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being  used  included  help  in  finding  data  and  help  in  interpreting  the  fault  logic  diagram. 
These  findings  demonstrate  that  the  FIND  approach  is  superior  to  the  conventional  paper- 
based  approach  in  terms  of  performance  efficiency  and  accuracy. 

5.4.4  Significant  Assistance  Required  in  Troubleshooting 

All  technicians  successfully  located  both  faults  without  significant  prompting  when 
working  with  FIND;  only  58  percent  of  the  technicians  found  the  fault  without  significant 
assistance  when  using  the  conventional  Technical  Manual.  This  finding  is  attributed  to  the 
proceduralization  and  consolidation  characteristics  of  FIND,  i.e.,  FIND  TI  prescribes 
which  tests  to  conduct  and  presents  all  the  information  needed  to  perform  these  tests  in  a 
small  number  of  text-graphic  modules.  By  contrast,  the  conventional  Technical  Manual 
requires  the  technician  to  choose  many  of  the  tests  and  to  work  with  separated  "how-to" 
information. 

5.4.5  Corrective  Maintenance  Performance  Time 

Overall,  the  mean  corrective  maintenance  task  performance  times  with  the 
different  types  of  TI  did  not  differ  significantly  from  one  another.  There  were  only  minor 
variations  at  the  subtask  level.  For  example,  the  slowest  performance  time  for 
experienced  technicians  was  for  removing  Circuit  Card  Assemblies  while  using  NTIPS 
electronically  delivered  TI  (16.5  minutes).  The  slowest  performance  time  for 
inexperienced  subjects  involved  removing  the  Circuit  Cards  with  the  conventional 
Technical  Manual  (16  minutes).  The  range  of  differences  in  performance  times  across  all 
subject  types  and  corrective  maintenance  tasks  was  5  minutes  41  seconds.  However,  the 
NTIPS  graphics  for  corrective  maintenance  TI  can  be  improved  considerably,  suggesting 
that  the  same  performance  improvement  obtained  with  FIND  can  be  obtained  with  the 
NTIPS  corrective  TI. 

5.4.6  Corrective  Maintenance  Errors 

All  technicians  completed  all  corrective  maintenance  tasks  successfully.  The  total 
numbers  of  minor  errors  made  in  carrying  out  corrective  maintenance  tasks  were,  with 
NTIPS  TI  in  paper  form  22;  with  NTIPS  TI  electronically  presented  27;  and  with 
conventional  Technical  Manual,  29.  The  largest  portion  of  these  errors  (approximately  63 
percent)  involved  failure  to  identify  or  locate  a  part.  In  most  cases  this  was  a  result  of 
inadequate  graphics  in  all  three  TI  types.  Overall,  experienced  technicians  made  fewer 
errors,  34  than  inexperienced  technicians,  44.  Again,  improvements  in  the  NTIPS  graphics 
should  result  in  significant  reduction  in  error  occurrence. 
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5.4.7  Graphics  Detail 

Many  of  the  graphics  in  the  NTIPS  TI  (especially  the  electronically  displayed 
version)  did  not  show  necessary  detail  and  were  too  small  to  be  of  real  benefit  to  the  user. 
The  quality  of  the  graphics  was  below  the  quality  obtainable  by  modern  state  of  the  art 
computer  systems.  This  defect  in  the  experimental  TI  resulted  in  an  unnecessarily  large 
number  of  inaccuracies  (looking  in  the  wrong  location  for  a  part  and  identifying  the  wrong 
part).  High  quality  graphics  are  required,  especially  in  terms  of  size,  for  future  electronic 
applications.  Ultimately,  optimization  of  text-graphics  modules  involves  trade-offs 
among  such  variables  as  (1)  the  spatial  relationships  of  text  to  graphics  on  a  given  display, 
(2)  the  determination  of  the  ideal  amount  of  work  prescribed  by  the  instructions  in  a 
single  frame,  (3)  the  resolution  of  the  display,  (4)  the  field-of-view  of  the  graphic,  and  (5) 
the  level  of  detail  provided  in  the  graphic.  Rules-of-thumb  and  conventions  based  on 
Human  Factors  Engineering  are  available  for  the  treatment  of  all  these  variables. 
Appropriate  guidance  must  be  incorporated  in  a  clearly  interpretable  form  into  future 
specifications  for  automated  generation  of  TI. 

5.4.8  Research  Issues 

A  number  of  short-range  changes  and  long-range  research  issues  were  identified 
during  the  field  test.  The  short  range  changes  which  will  have  the  greatest  impact  on 
system  performance  include  (1)  decreasing  response  time  for  FIND  and  (2)  increasing  size 
and  resolution  of  graphics  in  the  corrective  maintenance  TI. 

Longer  range  research  issues  include 

o  Making  use  of  animation  in  presenting  waveforms  and  other  change¬ 
able  data 

o  Presenting  schematics  and  signal  flow  diagrams  as  part  of  the  NTIPS 
troubleshooting  TI 

o  Providing  a  more  interactive,  user-directed  system  for  test 
sequencing 

o  Exploring  the  implications  of  smart  software  capable  of  profiting  by 
maintenance  experience. 
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APPENDIX  A 

DATA  COLLECTION  FORMS 


Troubleshooting  Form 
Corrective  Maintenance  Form 
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Faulted  Unit: Sweep  Beaolver  1A7A1B1 
TI:  electronic  Task:  Verification 
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TROUBT.F.STIOOTING  FORM 


Test  Point 


4-2.1  Task  1:  Ramova  Tuba  Focusing  Coll  113 

Subtasfc  2:  Ramova  Varlabla  Daisy  Una  T1  Type 


CORRECTIVE  MAINTENANCE  FORM 
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PREFERENCE  QUESTIONNAIRE  FOR  TROUBLE-SHOOTING 
Instructions  for  Completing  Questionnaire 

Now  that  you  have  done  troubleshooting  with  different  forms  of  technical 
information  (TI),  we  are  interested  in  your  evaluation.  First,  read  the  information  on  this 
page.  Then  complete  the  questionnaires  that  follow.  Be  sure  to  complete  every  item. 
Don't  leave  any  items  blank. 

Your  questionnaire  responses  will  not  be  used  to  rate  your  fitness.  All  of  your 
responses  will  be  kept  in  total  confidence. 

Some  questions  ask  you  to  choose  which  type  of  TI  (FIND  or  Conventional)  was 
better  for  a  particular  feature.  Other  questions  ask  you  to  rate  electronic  TI  features  on 
a  five-point  scale  from  1  (for  Very  Poor)  to  5  (for  Excellent).  Use  the  definitions  of  the 
numbers  in  the  scales  that  are  given  below  to  help  make  your  rating  decisions. 


1  - 

VERY  POOR 

I  don't  see  how  the  job  can  be  done  with  this  feature  the  way  it 
is. 

2  - 

POOR 

This  feature  isn't  very  good. 

3  - 

AVERAGE 

This  feature  is  O.K. 

4  - 

GOOD 

This  feature  makes  tasks  easier/quicker  to  perform. 

5  - 

EXCELLENT 

This  feature  is  really  great. 

As  you  come  to  the  lists  of  features  on  the  questionnaires,  try  to  remember  how 
much  each  feature  helped  or  hindered  you.  Select  the  rating  that  corresponds  to  your 
judgment,  and  mark  it  on  the  questionnaire. 

Use  the  COMMENTS  column  at  the  right  of  rating  scales  or  at  the  end  of  it  to  note 
any  strong  feelings  about  a  feature  or  to  suggest  how  it  might  be  improved. 

After  you  have  completed  the  rating  section  you  will  be  given  a  list  of  all  features 
of  electronic  delivery  and  asked  to  check  the  three  best  and  the  three  worst.  Be  sure  to 
complete  the  BIOGRAPHICAL  section  at  the  end  of  the  questionnaire. 

If  you  have  any  questions,  ask  a  data  collector  for  help. 
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1.  Check  the  type  of  TI  which  was  superior  on  each  characteristic. 


Troubleshooting  TI 

Information 

Characteristics  FIND  Conventional 


FAULT  VERIFICATION 

a.  Clarity  of  symptom  questions 

b.  Completeness  of  sympton  questions 

c.  Ease  of  initiating  fault  isolation 

steps 

d.  Presentation  format 
Fault  Isolation 

a.  Ease  of  selecting  initial  and 

next  tests 

b.  Ease  of  card  location 

c.  Ease  of  test  point  location 

d.  Ease  of  interpretation  of 

test  results 
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2.  Please  rate  the  following  electronic  system  features 


ELECTRONIC  SYSTEM  FEATURES 
FIND 


Electronic  System  Features 


Scale  Values 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


Techniques  for  Controlling 

Information  Delivery 

1.  Ease  of  using  menus  to  obtain 
maintenance  information 

2.  Ease  of  returning  to  the 
appropriate  section  in  a  set  of 
procedures  after  branching  to 
obtain  additional  information 

3.  Adequacy  of  features  to  exit 
from  an  inappropriate  section  of 
the  data  base  (e.g.,  following  an 
incorrect  key  press  or  equipment 
malfunction) 

4.  Adequacy  of  "prompts"  on  the 
display  for  assisting/guiding  the 
operator 

Features  of  FIND  TI 

1.  Format  of  Text 

2.  Level  of  procedure  detail 

3.  Sequence  of  tests 

4.  Validity  of  tests 

5.  Legibility  of  graphics 

6.  Understandability  of  graphics 

7.  Size  of  graphics 
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Features  of  Touch-Sensitive  Screen 
Operation 

1.  Arrangement  of  touch  labels 

2.  Location  of  touch  labels 

3.  Readability  of  touch  labels 

4.  Responsiveness  of  system  to 
using  touch  labels 


3.  Please  check  the  three  Best  and  three  Worst  characteristics  of  the  electronic 
information  delivery  system  you  used  for  Troubleshooting. 

BEST  AND  WORST  CHARACTERISTICS  OF  ELECTRONIC  DELIVERY 
OF  TECHNICAL  INFORMATION 

Characteristic  Best  Worst 

1.  Step  by  step  text 

2.  Organization  of  procedures 

3.  Relation  of  text  to  graphics 

4.  Size  of  text  characters 

5.  Font  (letter  style) 

6.  Spacing  and  layout 

7.  Size  of  graphics 

8.  Number  of  graphics 

9.  Nearness  of  graphic  to  related  text 

10.  Detail  of  graphic 

11.  Graphic  callouts 

12.  Touchscreen 

13.  Size  of  touch  boxes 

14.  Dependability  of  touch 

15.  Size  of  the  electronic  display 

16.  Electronic  display  legibility 

17.  Electronic  display  brightness 

18.  Electronic  display  glare 

19.  Printer 

20.  Menus 

21.  System  response  time 

22.  Ability  to  move  around  in  the  data  base 

Thanks  a  lot.  If  we  missed  any  characteristics  that  you  believe  to  be  important,  pro 
or  con,  jot  them  down  below. 
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PREFERENCE  QUESTIONNAIRE  FOR  CORRECTIVE  MAINTENANCE 


Now  that  you  have  completed  Corrective  Maintenance  with  three  different  types  of 
TI  (NTIPS  (E),  NTIPS  (P),  Conventional  paper),  we  are  interested  in  your  evaluation. 

Some  of  the  questionnaire  items  ask  you  to  rank  the  three  TI  types  on  various 
features  (1  being  most  preferred);  others  ask  for  ratings  on  a  five  point  scale.  As  with  the 
troubleshooting  questionnaire  you  are  also  asked  to  indicate  the  three  best  and  the  three 
worst  characteristics  of  electronic  delivery  as  it  relates  to  the  presentation  of  Corrective 
Maintenance  TI. 

Your  questionnaire  responses  will  not  be  used  to  rate  your  fitness.  All  your 
responses  will  be  kept  in  total  confidence. 
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1.  Please  rate  the  TI  you  used  for  setting  up  the  corrective  maintenance  process. 

TECHNICAL  INFORMATION 
Type  of  TI:  NTIPS(P)j  NTIPS(E);  Conventional 

Information  Characteristics  Strength  of  Approval  or  Disapproval 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 

CORRECTIVE  MAINTENANCE  TI 

1.  Introductory  Discussion  _  _  _  _  _  _ 

2.  Setup  Instructions 

a.  "Applicable  Configuration"  _  _  _  _  _  _ 

b.  Test  Equipment  _  _  _  _  _  _ 

c.  Tools  _  _  _  _  _  _ 

d.  Materials/Parts  List  _  _  _  _  _  _ 

e.  Task  References  _  _  _  _  _  _ 

f.  Personnel  Required  _  _  _  _  _  _ 

g.  Special  Skills  and  Knowledges  _  _  _  _  _  _ 

h.  Approximate  Time  Required  _  _  _ _ _  _ 

i.  List  of  Directives  _  _  _  _  _  _ 

3.  General  Safety  Instructions  _  _  _  _  _  _ 

SUPPORT  TI 

1.  Table  of  Contents  _  _  _  _  _  _ 

2.  IPB  _  _  _  _  _  _ 

3.  Preparatory  Instructions  _  _  _  _  _  _ 

a.  Indicator  Preparation  _  _  _  _  _  _ 

b.  Task  Preparation  _  _  _  _  _  _ 

4.  Others  (describe) 
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2.  Please  rank  the  three  types  of  TI  for  each  characteristic. 

Corrective  Maintenance  TI 

NTIPS  NTIPS 

(E)  (P) 

1.  Step  Instructions 

a.  Organization  into  Tasks,  _  _ 

Subtasks,  and  Steps 

b.  Amount  of  text  _  _ 

c.  Usability  of  text  _  _ 

-  Level  of  Detail  _  _ 

-  Format  _ 

-  Clarity  of  Writing  _  _ 

2.  Graphics 

a.  Amount  of  Graphics  _  _ 

b.  Usability  of  Graphics  _  _ 

-  Legibility  _  _ 

-  Understandability  _  _ 

-  Size  _  _ 

-  Ease  in  Finding  Components  _  _ 

-  Level  of  Detail  _  _ 

-  Format  _  _ 


Information 

Characteristics 


Conventional 
Work  Package 
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3. 


Please  rate  the  following  electronic  system  features. 


ELECTRONIC  SYSTEM  FEATURES 
CORRECTIVE  MAINTENANCE 


Electronic  System  Features 


Scale  Values 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc 


Techniques  for  Controlling 

Information  Delivery 

1.  Ease  of  using  menus  to  obtain 
maintenance  information 

2.  Ease  of  returning  to  the 
appropriate  section  in  a  set  of 
procedures  after  branching  to 
obtain  additional  information 

3.  Adequacy  of  features  to  exit 
from  an  inappropriate  section  of 
the  data  base  (e.g.,  following  an 
incorrect  key  press  or  equipment 
malfunction) 

4.  Adequacy  of  "prompts"  on  the 
display  for  assisting/guiding  the 
operator 

Features  of  Touch-Sensitive  Screen 

Operation 

1.  Arrangement  of  touch  labels 

2.  Location  of  touch  labels 

3.  Readability  of  touch  labels 

4.  Responsiveness  of  system  to 
using  touch  labels 


Comments 
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4.  Please  check  the  three  Best  and  three  Worst  characteristics  of  the  electronic 
information  delivery  system  you  used  for  Corrective  Maintenance. 

BEST  AND  WORST  CHARACTERISTICS  OF  ELECTRONIC  DELIVERY 
OF  TECHNICAL  INFORMATION 

Characteristic  Best  Worst 

1.  Step  by  step  text 

2.  Organization  of  procedures 

3.  Relation  of  text  to  graphics 

4.  Size  of  text  characters 

5.  Font  (letter  style) 

6.  Spacing  and  layout 

7.  Size  of  graphics 

8.  Number  of  graphics 

9.  Nearness  of  graphic  to  related  text 

10.  Detail  of  graphic 

11.  Graphic  callouts 

12.  Touch  screen 

13.  Size  of  touch  boxes 

14.  Dependability  of  touch 

15.  Size  of  the  electronic  display 

16.  Electronic  display  legibility 

17.  Electronic  display  brightness 

18.  Electronic  display  glare 

19.  Printer 

20.  Menus 

21.  System  response  time 

22.  Ability  to  move  around  in  the  data  base 

Thanks  a  lot.  If  we  missed  any  characteristics  that  you  believe  to  be  important,  pro 
or  con,  jot  them  down  below. 


S.  Please  complete  the  following  questions  based  on  your  experience  with  both 
troubleshooting  and  corrective  maintenance  TI. 


ELECTRONIC  SYSTEM  FEATURES:  GENERAL 


Electronic  System  Features  Scale  Values 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


Features  of  NTIPS  Keys  and 

Keyboard 

1.  Spacing  of  keys 

2.  Arrangement  of  keys 

3.  Ease  of  operating  keys 

4.  Indication(s)  that  keys  have  been 
activated 

5.  Reliability  of  keys  (i.e.,  how  well 
did  the  keys  respond  to  use) 

General  Screen  Features 

1.  Adequacy  of  screen  size  for 
display  of  information 

2.  Brightness  of  display 

3.  Readability  of  display  screens 

4.  Contrast  between  displayed 
information  and  background 

5.  Glare  resistance  of  display 
screen 
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GENERAL  COMMENTS 


If  you  had  a  choice  of  using  an  electronic  or  paper-based  manual  to  perform  tasks 
which  would  you  choose?  _ 

In  working  with  the  technical  documentation,  was  it  easier  with  the  electronic 
device? _  Or  paper? _ 

Which  mode  of  presentation  was  better  organized  for  your  purposes? 
Electronic? _  Or  paper? _ 

What  do  you  see  as  the  major  advantages  of  the  electronic  presentation? 


The  paper  presentation? 


APPENDIX  C 

EXAMPLES  OF  TECHNICAL  INFORMATION 
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FIND  Fault  Verification  Procedures  72 
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FIND  FAULT  VERIFICATION  PROCEDURES 


72 


FIND  CARD  LOCATOR 
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FIND  TEST  PROCEDURES 


CONVENTIONAL  TECHNICAL  MANUAL 
FAULT  LOGIC  DIAGRAM 


CONVENTIONAL  TECHNICAL  MANUAL 
SIGNAL  FLOW  DIAGRAM  (PARTIAL) 


NTIPS  ELECTRONIC  —  CORRECTIVE  MAINTENANCE 

(SCREEN  PRINT) 
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NTIPS  PAPER  —  CORRECTIVE  MAINTENANCE 


Gopy  avaiiall,  to  DTIC  t”‘l 

p*rmit  hill?  legible  i«piodb^teh 
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driver  amplifier  1A14A1 

2.  Looeen  two  cap  acrews  (2)  in  coupling  clamp  (3) 

3.  Remove  aix  acrawa  (4).  lockwaahara  (5).  and  Hat 
waahera  (6)  Ramova  variable  daiay  lina  1A14A2 


CONVENTIONAL  PAPER:  CROSS  REFERENCE 

AND  PAGE-TURNING 
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